[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:19:24 dist-test-slave-b83v krb5kdc[3151](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:19:24 dist-test-slave-b83v krb5kdc[3151](info): set up 2 sockets
Sep 28 20:19:24 dist-test-slave-b83v krb5kdc[3151](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 20:19:25.340601 3144 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Sep 28 20:19:25 dist-test-slave-b83v krb5kdc[3151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626765, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:19:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:19:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.18.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab.
Entry for principal kudu/127.3.18.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab.
I20260928 20:19:25.472257 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:39003
--webserver_interface=127.3.18.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:36783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.18.62:39003
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:19:25.557598 3164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:19:25.557822 3164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:19:25.557847 3164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:19:25.558990 3164 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:19:25.559031 3164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:19:25.559046 3164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:19:25.559057 3164 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:19:25.559069 3164 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:19:25.560626 3164 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:36783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.18.62:39003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:39003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.18.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:19:25.560855 3164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:19:25.561043 3164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 20:19:25.566568 3164 server_base.cc:1034] running on GCE node
W20260928 20:19:25.566525 3172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:25.566562 3169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:25.566525 3170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:25.567058 3164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:19:25.567397 3164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:19:25.568583 3164 hybrid_clock.cc:648] HybridClock initialized: now 1790626765568568 us; error 32 us; skew 500 ppm
Sep 28 20:19:25 dist-test-slave-b83v krb5kdc[3151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626765, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:19:25.577901 3164 init.cc:377] Logged in from keytab as kudu/127.3.18.62@KRBTEST.COM (short username kudu)
I20260928 20:19:25.578485 3164 webserver.cc:460] Webserver started at http://127.3.18.62:33243/ using document root <none> and password file <none>
I20260928 20:19:25.578758 3164 fs_manager.cc:359] Metadata directory not provided
I20260928 20:19:25.578826 3164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:19:25.578971 3164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:19:25.580228 3164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/minicluster-data/master-0/data/instance:
uuid: "dc129e389f1d41e796a7139d9187b147"
format_stamp: "Formatted at 2026-09-28 20:19:25 on dist-test-slave-b83v"
server_key: "b24118fa44996d8484caa0c91a7bca35"
server_key_iv: "048ec33a00ab399b874b19eac16166a5"
server_key_version: "encryptionkey@0"
I20260928 20:19:25.580627 3164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/minicluster-data/master-0/wal/instance:
uuid: "dc129e389f1d41e796a7139d9187b147"
format_stamp: "Formatted at 2026-09-28 20:19:25 on dist-test-slave-b83v"
server_key: "b24118fa44996d8484caa0c91a7bca35"
server_key_iv: "048ec33a00ab399b874b19eac16166a5"
server_key_version: "encryptionkey@0"
I20260928 20:19:25.582229 3164 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 20:19:25.583099 3179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:25.583364 3164 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:19:25.583468 3164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/minicluster-data/master-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/minicluster-data/master-0/wal
uuid: "dc129e389f1d41e796a7139d9187b147"
format_stamp: "Formatted at 2026-09-28 20:19:25 on dist-test-slave-b83v"
server_key: "b24118fa44996d8484caa0c91a7bca35"
server_key_iv: "048ec33a00ab399b874b19eac16166a5"
server_key_version: "encryptionkey@0"
I20260928 20:19:25.583581 3164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:19:25.599464 3164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:19:25.602147 3164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:19:25.602330 3164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:19:25.606891 3164 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.62:39003
I20260928 20:19:25.607018 3231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.62:39003 every 8 connection(s)
I20260928 20:19:25.607331 3164 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
I20260928 20:19:25.608158 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 3164
I20260928 20:19:25.608275 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/minicluster-data/master-0/wal/instance
I20260928 20:19:25.608515 3232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:25.609185 3144 external_mini_cluster.cc:1428] Setting key 986b32d06eb347aeaee08ae33051e01f
I20260928 20:19:25.611927 3232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc129e389f1d41e796a7139d9187b147: Bootstrap starting.
I20260928 20:19:25.612679 3232 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc129e389f1d41e796a7139d9187b147: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:25.613016 3232 log.cc:824] T 00000000000000000000000000000000 P dc129e389f1d41e796a7139d9187b147: Log is configured to *not* fsync() on all Append() calls
I20260928 20:19:25.613857 3232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc129e389f1d41e796a7139d9187b147: No bootstrap required, opened a new log
I20260928 20:19:25.616657 3232 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc129e389f1d41e796a7139d9187b147 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc129e389f1d41e796a7139d9187b147" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 39003 } }
I20260928 20:19:25.616799 3232 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc129e389f1d41e796a7139d9187b147 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:25.616822 3232 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc129e389f1d41e796a7139d9187b147 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc129e389f1d41e796a7139d9187b147, State: Initialized, Role: FOLLOWER
I20260928 20:19:25.616961 3232 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc129e389f1d41e796a7139d9187b147 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc129e389f1d41e796a7139d9187b147" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 39003 } }
I20260928 20:19:25.617044 3232 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc129e389f1d41e796a7139d9187b147 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:19:25.617069 3232 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc129e389f1d41e796a7139d9187b147 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:19:25.617137 3232 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc129e389f1d41e796a7139d9187b147 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:25.617830 3232 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc129e389f1d41e796a7139d9187b147 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc129e389f1d41e796a7139d9187b147" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 39003 } }
I20260928 20:19:25.617945 3232 leader_election.cc:302] T 00000000000000000000000000000000 P dc129e389f1d41e796a7139d9187b147 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc129e389f1d41e796a7139d9187b147; no voters:
I20260928 20:19:25.618129 3232 leader_election.cc:288] T 00000000000000000000000000000000 P dc129e389f1d41e796a7139d9187b147 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:19:25.618187 3237 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc129e389f1d41e796a7139d9187b147 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:19:25.618368 3237 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc129e389f1d41e796a7139d9187b147 [term 1 LEADER]: Becoming Leader. State: Replica: dc129e389f1d41e796a7139d9187b147, State: Running, Role: LEADER
I20260928 20:19:25.618407 3232 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc129e389f1d41e796a7139d9187b147 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:19:25.618460 3237 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc129e389f1d41e796a7139d9187b147 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc129e389f1d41e796a7139d9187b147" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 39003 } }
I20260928 20:19:25.618819 3237 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc129e389f1d41e796a7139d9187b147 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc129e389f1d41e796a7139d9187b147. Latest consensus state: current_term: 1 leader_uuid: "dc129e389f1d41e796a7139d9187b147" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc129e389f1d41e796a7139d9187b147" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 39003 } } }
I20260928 20:19:25.618885 3237 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc129e389f1d41e796a7139d9187b147 [sys.catalog]: This master's current role is: LEADER
I20260928 20:19:25.619164 3238 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc129e389f1d41e796a7139d9187b147 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc129e389f1d41e796a7139d9187b147" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc129e389f1d41e796a7139d9187b147" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 39003 } } }
I20260928 20:19:25.619246 3238 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc129e389f1d41e796a7139d9187b147 [sys.catalog]: This master's current role is: LEADER
I20260928 20:19:25.619375 3243 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:19:25.619917 3243 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:19:25.621584 3243 catalog_manager.cc:1347] Generated new cluster ID: c25e0b9d2b5a4941ac73871d63f0c87d
I20260928 20:19:25.621628 3243 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 20:19:25 dist-test-slave-b83v krb5kdc[3151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626765, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:19:25.628191 3235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:25.610215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58872 (local address 127.3.18.62:39003)
0928 20:19:25.610408 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:25.610419 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:25.610692 (+ 273us) server_negotiation.cc:407] Connection header received
0928 20:19:25.613739 (+ 3047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:25.613746 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:25.613796 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:25.613914 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:25.616268 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:25.617420 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:25.619048 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:25.619719 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:25.619877 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:25.623996 (+ 4119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:25.624000 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:25.624001 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:25.624014 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:25.626083 (+ 2069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:25.626640 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:25.626644 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:25.626645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:25.626747 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:25.627041 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:25.627042 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:25.627042 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:25.627331 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:25.627486 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:25.627840 (+ 354us) server_negotiation.cc:299] Negotiation successful
0928 20:19:25.627979 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":83,"threads_started":1}
I20260928 20:19:25.633667 3243 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:19:25.634289 3243 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:19:25.638406 3243 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc129e389f1d41e796a7139d9187b147: Generated new TSK 0
I20260928 20:19:25.638559 3243 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.18.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab.
Entry for principal kudu/127.3.18.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab.
I20260928 20:19:25.671880 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.1:0
--local_ip_for_outbound_sockets=127.3.18.1
--webserver_interface=127.3.18.1
--webserver_port=0
--tserver_master_addrs=127.3.18.62:39003
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab
--principal=kudu/127.3.18.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:36783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:19:25.756821 3258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:19:25.757089 3258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:19:25.757133 3258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:19:25.758414 3258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:19:25.758499 3258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:19:25.758546 3258 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.1
I20260928 20:19:25.760221 3258 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:36783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab
--principal=kudu/127.3.18.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.18.1
--webserver_port=0
--tserver_master_addrs=127.3.18.62:39003
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.1
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:19:25.760532 3258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:19:25.760788 3258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:19:25.766464 3266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:25.766453 3263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:25.766700 3258 server_base.cc:1034] running on GCE node
W20260928 20:19:25.766456 3264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:25.766973 3258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:19:25.767205 3258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:19:25.768419 3258 hybrid_clock.cc:648] HybridClock initialized: now 1790626765768403 us; error 42 us; skew 500 ppm
Sep 28 20:19:25 dist-test-slave-b83v krb5kdc[3151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626765, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:19:25.770977 3258 init.cc:377] Logged in from keytab as kudu/127.3.18.1@KRBTEST.COM (short username kudu)
I20260928 20:19:25.771401 3258 webserver.cc:460] Webserver started at http://127.3.18.1:38629/ using document root <none> and password file <none>
I20260928 20:19:25.771615 3258 fs_manager.cc:359] Metadata directory not provided
I20260928 20:19:25.771657 3258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:19:25.771746 3258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:19:25.772823 3258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/minicluster-data/ts-0/data/instance:
uuid: "379254282e524812952df84afd6a001f"
format_stamp: "Formatted at 2026-09-28 20:19:25 on dist-test-slave-b83v"
server_key: "f80f2ca65c4166f164619dd5181d9924"
server_key_iv: "6be61236119687ef429e55a02af8c60c"
server_key_version: "encryptionkey@0"
I20260928 20:19:25.773128 3258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/minicluster-data/ts-0/wal/instance:
uuid: "379254282e524812952df84afd6a001f"
format_stamp: "Formatted at 2026-09-28 20:19:25 on dist-test-slave-b83v"
server_key: "f80f2ca65c4166f164619dd5181d9924"
server_key_iv: "6be61236119687ef429e55a02af8c60c"
server_key_version: "encryptionkey@0"
I20260928 20:19:25.774478 3258 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 20:19:25.775326 3273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:25.775544 3258 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:19:25.775635 3258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/minicluster-data/ts-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/minicluster-data/ts-0/wal
uuid: "379254282e524812952df84afd6a001f"
format_stamp: "Formatted at 2026-09-28 20:19:25 on dist-test-slave-b83v"
server_key: "f80f2ca65c4166f164619dd5181d9924"
server_key_iv: "6be61236119687ef429e55a02af8c60c"
server_key_version: "encryptionkey@0"
I20260928 20:19:25.775688 3258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:19:25.783082 3258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:19:25.785480 3258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:19:25.785604 3258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:19:25.785795 3258 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:19:25.786269 3258 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:19:25.786309 3258 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:25.786334 3258 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:19:25.786346 3258 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:25.792495 3258 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.1:39571
I20260928 20:19:25.792568 3386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.1:39571 every 8 connection(s)
I20260928 20:19:25.792906 3258 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
I20260928 20:19:25.797370 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 3258
I20260928 20:19:25.797468 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/minicluster-data/ts-0/wal/instance
I20260928 20:19:25.797627 3144 external_mini_cluster.cc:1428] Setting key d225068c766b4cdb4e4bb7ff3237b30e
Sep 28 20:19:25 dist-test-slave-b83v krb5kdc[3151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626765, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.1@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:19:25.803478 3235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:25.793668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:52685 (local address 127.3.18.62:39003)
0928 20:19:25.793720 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:25.793723 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:25.793952 (+ 229us) server_negotiation.cc:407] Connection header received
0928 20:19:25.794438 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:25.794439 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:25.794450 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:25.794506 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:25.795597 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:25.796208 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:25.797224 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:25.797611 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:25.797706 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:25.800390 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:25.800392 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:25.800392 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:25.800400 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:25.801863 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:25.802386 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:25.802387 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:25.802387 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:25.802444 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:25.802752 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:25.802752 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:25.802753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:25.802863 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:25.802916 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:25.803319 (+ 403us) server_negotiation.cc:299] Negotiation successful
0928 20:19:25.803403 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 20:19:25.803830 3389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:25.793667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:39003 (local address 127.3.18.1:52685)
0928 20:19:25.793832 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:25.793868 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:25.794351 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:25.794595 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:25.794599 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:25.795263 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:25.795461 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:25.795467 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:25.796295 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:25.796296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:25.797127 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:25.797129 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:25.797744 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:25.797745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:25.797912 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:25.798569 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:19:25.798578 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:25.800242 (+ 1664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:25.801948 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:25.801957 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:25.801960 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:25.802307 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:25.802517 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:25.802518 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:25.802519 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:25.802678 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:25.802928 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:25.802935 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:25.803144 (+ 209us) client_negotiation.cc:769] Sending connection context
0928 20:19:25.803280 (+ 136us) client_negotiation.cc:240] Negotiation successful
0928 20:19:25.803482 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":90,"threads_started":1}
I20260928 20:19:25.804463 3387 heartbeater.cc:347] Connected to a master server at 127.3.18.62:39003
I20260928 20:19:25.804605 3387 heartbeater.cc:464] Registering TS with master...
I20260928 20:19:25.804812 3387 heartbeater.cc:511] Master 127.3.18.62:39003 requested a full tablet report, sending...
I20260928 20:19:25.805447 3196 ts_manager.cc:194] Registered new tserver with Master: 379254282e524812952df84afd6a001f (127.3.18.1:39571)
I20260928 20:19:25.806191 3196 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.18.1@KRBTEST.COM'} at 127.3.18.1:52685
WARNING: no policy specified for kudu/127.3.18.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab.
Entry for principal kudu/127.3.18.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab.
I20260928 20:19:25.828501 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.2:0
--local_ip_for_outbound_sockets=127.3.18.2
--webserver_interface=127.3.18.2
--webserver_port=0
--tserver_master_addrs=127.3.18.62:39003
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab
--principal=kudu/127.3.18.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:36783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:19:25.914156 3392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:19:25.914376 3392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:19:25.914400 3392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:19:25.915582 3392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:19:25.915628 3392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:19:25.915656 3392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.2
I20260928 20:19:25.917153 3392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:36783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab
--principal=kudu/127.3.18.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.18.2
--webserver_port=0
--tserver_master_addrs=127.3.18.62:39003
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.2
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:19:25.917405 3392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:19:25.917651 3392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:19:25.923161 3398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:25.923215 3400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:25.923231 3397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:25.923475 3392 server_base.cc:1034] running on GCE node
I20260928 20:19:25.923717 3392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:19:25.924057 3392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:19:25.925242 3392 hybrid_clock.cc:648] HybridClock initialized: now 1790626765925228 us; error 42 us; skew 500 ppm
Sep 28 20:19:25 dist-test-slave-b83v krb5kdc[3151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626765, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:19:25.928184 3392 init.cc:377] Logged in from keytab as kudu/127.3.18.2@KRBTEST.COM (short username kudu)
I20260928 20:19:25.928717 3392 webserver.cc:460] Webserver started at http://127.3.18.2:41105/ using document root <none> and password file <none>
I20260928 20:19:25.928966 3392 fs_manager.cc:359] Metadata directory not provided
I20260928 20:19:25.929037 3392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:19:25.929195 3392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:19:25.930380 3392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/minicluster-data/ts-1/data/instance:
uuid: "0d502e791f8141ff94aea82b14c63f49"
format_stamp: "Formatted at 2026-09-28 20:19:25 on dist-test-slave-b83v"
server_key: "c4d12659d9f3e84a420ad6b1b106c075"
server_key_iv: "caed46eba22f9e484202a54f5c1baf74"
server_key_version: "encryptionkey@0"
I20260928 20:19:25.930788 3392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/minicluster-data/ts-1/wal/instance:
uuid: "0d502e791f8141ff94aea82b14c63f49"
format_stamp: "Formatted at 2026-09-28 20:19:25 on dist-test-slave-b83v"
server_key: "c4d12659d9f3e84a420ad6b1b106c075"
server_key_iv: "caed46eba22f9e484202a54f5c1baf74"
server_key_version: "encryptionkey@0"
I20260928 20:19:25.932482 3392 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:19:25.933293 3407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:25.933533 3392 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:19:25.933640 3392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/minicluster-data/ts-1/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/minicluster-data/ts-1/wal
uuid: "0d502e791f8141ff94aea82b14c63f49"
format_stamp: "Formatted at 2026-09-28 20:19:25 on dist-test-slave-b83v"
server_key: "c4d12659d9f3e84a420ad6b1b106c075"
server_key_iv: "caed46eba22f9e484202a54f5c1baf74"
server_key_version: "encryptionkey@0"
I20260928 20:19:25.933724 3392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:19:25.953701 3392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:19:25.956349 3392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:19:25.956676 3392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:19:25.957075 3392 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:19:25.957494 3392 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:19:25.957556 3392 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:25.957597 3392 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:19:25.957618 3392 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:25.963946 3392 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.2:39021
I20260928 20:19:25.964018 3522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.2:39021 every 8 connection(s)
I20260928 20:19:25.964341 3392 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
Sep 28 20:19:25 dist-test-slave-b83v krb5kdc[3151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626765, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.2@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:19:25.974236 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 3392
I20260928 20:19:25.974350 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/minicluster-data/ts-1/wal/instance
I20260928 20:19:25.974583 3144 external_mini_cluster.cc:1428] Setting key eefb0c73f3d9c2606820fc9b9b2cea5f
I20260928 20:19:25.974781 3235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:25.965280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:53725 (local address 127.3.18.62:39003)
0928 20:19:25.965363 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:25.965365 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:25.965551 (+ 186us) server_negotiation.cc:407] Connection header received
0928 20:19:25.966068 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:25.966070 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:25.966080 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:25.966149 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:25.967176 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:25.967827 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:25.968910 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:25.969123 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:25.969202 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:25.971427 (+ 2225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:25.971428 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:25.971429 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:25.971436 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:25.972957 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:25.973414 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:25.973415 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:25.973415 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:25.973470 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:25.973751 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:25.973751 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:25.973752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:25.973862 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:25.973917 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:25.974518 (+ 601us) server_negotiation.cc:299] Negotiation successful
0928 20:19:25.974672 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 20:19:25.974929 3525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:25.965250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:39003 (local address 127.3.18.2:53725)
0928 20:19:25.965412 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:25.965455 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:25.965955 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:25.966208 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:25.966211 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:25.966827 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:25.967041 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:25.967048 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:25.967939 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:25.967940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:25.968809 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:25.968812 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:25.969220 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:25.969221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:25.969337 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:25.969954 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:19:25.969962 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:25.971340 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:25.973059 (+ 1719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:25.973069 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:25.973072 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:25.973340 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:25.973549 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:25.973550 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:25.973550 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:25.973691 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:25.974064 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:25.974072 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:25.974268 (+ 196us) client_negotiation.cc:769] Sending connection context
0928 20:19:25.974468 (+ 200us) client_negotiation.cc:240] Negotiation successful
0928 20:19:25.974676 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":85,"threads_started":1}
I20260928 20:19:25.975548 3523 heartbeater.cc:347] Connected to a master server at 127.3.18.62:39003
I20260928 20:19:25.975646 3523 heartbeater.cc:464] Registering TS with master...
I20260928 20:19:25.975873 3523 heartbeater.cc:511] Master 127.3.18.62:39003 requested a full tablet report, sending...
I20260928 20:19:25.976276 3196 ts_manager.cc:194] Registered new tserver with Master: 0d502e791f8141ff94aea82b14c63f49 (127.3.18.2:39021)
I20260928 20:19:25.976657 3196 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.18.2@KRBTEST.COM'} at 127.3.18.2:53725
WARNING: no policy specified for kudu/127.3.18.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab.
Entry for principal kudu/127.3.18.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab.
I20260928 20:19:26.004030 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.3:0
--local_ip_for_outbound_sockets=127.3.18.3
--webserver_interface=127.3.18.3
--webserver_port=0
--tserver_master_addrs=127.3.18.62:39003
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab
--principal=kudu/127.3.18.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:36783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:19:26.089496 3528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:19:26.089715 3528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:19:26.089740 3528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:19:26.090963 3528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:19:26.091014 3528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:19:26.091043 3528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.3
I20260928 20:19:26.092634 3528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:36783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab
--principal=kudu/127.3.18.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.18.3
--webserver_port=0
--tserver_master_addrs=127.3.18.62:39003
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.3
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:19:26.092904 3528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:19:26.093175 3528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:19:26.098501 3533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:26.098661 3528 server_base.cc:1034] running on GCE node
W20260928 20:19:26.098529 3536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:26.098774 3534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:26.098986 3528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:19:26.099304 3528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:19:26.100461 3528 hybrid_clock.cc:648] HybridClock initialized: now 1790626766100456 us; error 52 us; skew 500 ppm
Sep 28 20:19:26 dist-test-slave-b83v krb5kdc[3151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626766, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:19:26.103014 3528 init.cc:377] Logged in from keytab as kudu/127.3.18.3@KRBTEST.COM (short username kudu)
I20260928 20:19:26.103616 3528 webserver.cc:460] Webserver started at http://127.3.18.3:45519/ using document root <none> and password file <none>
I20260928 20:19:26.103935 3528 fs_manager.cc:359] Metadata directory not provided
I20260928 20:19:26.104084 3528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:19:26.104244 3528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:19:26.105361 3528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/minicluster-data/ts-2/data/instance:
uuid: "791c294467384dcfb0b56ac0ef14917b"
format_stamp: "Formatted at 2026-09-28 20:19:26 on dist-test-slave-b83v"
server_key: "cce4204296e6118779b9fb30ca9c7063"
server_key_iv: "a43f27064c3b10be3cca78286437ffa7"
server_key_version: "encryptionkey@0"
I20260928 20:19:26.105749 3528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/minicluster-data/ts-2/wal/instance:
uuid: "791c294467384dcfb0b56ac0ef14917b"
format_stamp: "Formatted at 2026-09-28 20:19:26 on dist-test-slave-b83v"
server_key: "cce4204296e6118779b9fb30ca9c7063"
server_key_iv: "a43f27064c3b10be3cca78286437ffa7"
server_key_version: "encryptionkey@0"
I20260928 20:19:26.107194 3528 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:19:26.108080 3544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:26.108281 3528 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:19:26.108378 3528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/minicluster-data/ts-2/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/minicluster-data/ts-2/wal
uuid: "791c294467384dcfb0b56ac0ef14917b"
format_stamp: "Formatted at 2026-09-28 20:19:26 on dist-test-slave-b83v"
server_key: "cce4204296e6118779b9fb30ca9c7063"
server_key_iv: "a43f27064c3b10be3cca78286437ffa7"
server_key_version: "encryptionkey@0"
I20260928 20:19:26.108480 3528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:19:26.130930 3528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:19:26.133881 3528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:19:26.134087 3528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:19:26.134367 3528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:19:26.134721 3528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:19:26.134775 3528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:26.134819 3528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:19:26.134850 3528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:26.142058 3528 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.3:44045
I20260928 20:19:26.142133 3657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.3:44045 every 8 connection(s)
I20260928 20:19:26.142505 3528 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
I20260928 20:19:26.149796 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 3528
I20260928 20:19:26.149926 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790626764250411-3144-0/minicluster-data/ts-2/wal/instance
Sep 28 20:19:26 dist-test-slave-b83v krb5kdc[3151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626766, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.3@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:19:26.150209 3144 external_mini_cluster.cc:1428] Setting key e6ce0a68bccc3bad5393d11ae0b65a49
I20260928 20:19:26.154407 3235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:26.143540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:54145 (local address 127.3.18.62:39003)
0928 20:19:26.143589 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:26.143592 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:26.143605 (+ 13us) server_negotiation.cc:407] Connection header received
0928 20:19:26.144243 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:26.144245 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:26.144258 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:26.144334 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:26.145553 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:26.146063 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:26.147381 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:26.147583 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:26.147654 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:26.150821 (+ 3167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:26.150823 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:26.150824 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:26.150834 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:26.152572 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:26.153085 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:26.153091 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:26.153091 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:26.153153 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:26.153443 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:26.153444 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:26.153445 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:26.153575 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:26.153865 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:26.154204 (+ 339us) server_negotiation.cc:299] Negotiation successful
0928 20:19:26.154308 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 20:19:26.154762 3660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:26.143315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:39003 (local address 127.3.18.3:54145)
0928 20:19:26.143484 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:26.143530 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:26.144113 (+ 583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:26.144473 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:26.144477 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:26.145184 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:26.145412 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:26.145418 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:26.146233 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:26.146234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:26.147259 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:26.147262 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:26.147698 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:26.147699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:26.148001 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:26.148653 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:19:26.148806 (+ 153us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:26.150658 (+ 1852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:26.152699 (+ 2041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:26.152709 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:26.152712 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:26.152983 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:26.153230 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:26.153231 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:26.153231 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:26.153369 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:26.153861 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:26.153870 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:26.154045 (+ 175us) client_negotiation.cc:769] Sending connection context
0928 20:19:26.154164 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 20:19:26.154354 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":96,"threads_started":1}
I20260928 20:19:26.155447 3658 heartbeater.cc:347] Connected to a master server at 127.3.18.62:39003
I20260928 20:19:26.155560 3658 heartbeater.cc:464] Registering TS with master...
I20260928 20:19:26.155885 3658 heartbeater.cc:511] Master 127.3.18.62:39003 requested a full tablet report, sending...
I20260928 20:19:26.156361 3196 ts_manager.cc:194] Registered new tserver with Master: 791c294467384dcfb0b56ac0ef14917b (127.3.18.3:44045)
I20260928 20:19:26.156806 3196 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.18.3@KRBTEST.COM'} at 127.3.18.3:54145
I20260928 20:19:26.163192 3144 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 20:19:26 dist-test-slave-b83v krb5kdc[3151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626766, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 20:19:26 dist-test-slave-b83v krb5kdc[3151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626766, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.18.1@KRBTEST.COM
I20260928 20:19:26.190289 3665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:26.179941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52892 (local address 127.3.18.1:39571)
0928 20:19:26.180178 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:26.180183 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:26.180196 (+ 13us) server_negotiation.cc:407] Connection header received
0928 20:19:26.180229 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:26.180231 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:26.180245 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:26.180330 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:26.181006 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:26.181807 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:26.182879 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:26.183294 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:26.183378 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:26.186413 (+ 3035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:26.186428 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:26.186428 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:26.186437 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:26.188612 (+ 2175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:26.189075 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:26.189079 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:26.189080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:26.189157 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:26.189517 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:26.189519 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:26.189519 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:26.189691 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:26.189835 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:26.190012 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 20:19:26.190097 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":104,"threads_started":1}
W20260928 20:19:26.190737 3301 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52892
Sep 28 20:19:26 dist-test-slave-b83v krb5kdc[3151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 20:19:26 dist-test-slave-b83v krb5kdc[3151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.18.1@KRBTEST.COM
I20260928 20:19:26.216050 3665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:26.207277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52908 (local address 127.3.18.1:39571)
0928 20:19:26.207347 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:26.207349 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:26.207358 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:19:26.207384 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:26.207385 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:26.207397 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:26.207453 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:26.208218 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:26.208737 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:26.209590 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:26.209835 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:26.209986 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:26.213073 (+ 3087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:26.213075 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:26.213075 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:26.213085 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:26.214820 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:26.215242 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:26.215244 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:26.215245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:26.215308 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:26.215517 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:26.215518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:26.215519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:26.215634 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:26.215760 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:26.215892 (+ 132us) server_negotiation.cc:299] Negotiation successful
0928 20:19:26.215981 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 20:19:26.216565 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 3258
I20260928 20:19:26.221609 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 3392
I20260928 20:19:26.226560 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 3528
I20260928 20:19:26.230985 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 3164
2026-09-28T20:19:26Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1988 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:19:26 dist-test-slave-b83v krb5kdc[3672](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:19:26 dist-test-slave-b83v krb5kdc[3672](info): set up 2 sockets
Sep 28 20:19:26 dist-test-slave-b83v krb5kdc[3672](info): commencing operation
krb5kdc: starting...
W20260928 20:19:27.283546 3144 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Sep 28 20:19:27 dist-test-slave-b83v krb5kdc[3672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:19:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:19:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.18.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab.
Entry for principal kudu/127.3.18.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab.
I20260928 20:19:27.411032 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:39501
--webserver_interface=127.3.18.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:40409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.18.62:39501
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:19:27.495698 3685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:19:27.495962 3685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:19:27.495990 3685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:19:27.497156 3685 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:19:27.497198 3685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:19:27.497212 3685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:19:27.497224 3685 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:19:27.497236 3685 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:19:27.498687 3685 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:40409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.18.62:39501
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:39501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.18.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:19:27.498895 3685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:19:27.499075 3685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:19:27.504117 3693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:27.504117 3690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:27.504179 3691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:27.504293 3685 server_base.cc:1034] running on GCE node
I20260928 20:19:27.504513 3685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:19:27.504801 3685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:19:27.506004 3685 hybrid_clock.cc:648] HybridClock initialized: now 1790626767505980 us; error 37 us; skew 500 ppm
Sep 28 20:19:27 dist-test-slave-b83v krb5kdc[3672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626767, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:19:27.508760 3685 init.cc:377] Logged in from keytab as kudu/127.3.18.62@KRBTEST.COM (short username kudu)
I20260928 20:19:27.509258 3685 webserver.cc:460] Webserver started at http://127.3.18.62:40915/ using document root <none> and password file <none>
I20260928 20:19:27.509490 3685 fs_manager.cc:359] Metadata directory not provided
I20260928 20:19:27.509552 3685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:19:27.509698 3685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:19:27.510823 3685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/minicluster-data/master-0/data/instance:
uuid: "b8ed57f8198445929fc6676620305d9c"
format_stamp: "Formatted at 2026-09-28 20:19:27 on dist-test-slave-b83v"
server_key: "83d91038ca5df2c14cc8e66526ec82e3"
server_key_iv: "9d353ebe66b70373fc6636c9ece96bc4"
server_key_version: "encryptionkey@0"
I20260928 20:19:27.511199 3685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/minicluster-data/master-0/wal/instance:
uuid: "b8ed57f8198445929fc6676620305d9c"
format_stamp: "Formatted at 2026-09-28 20:19:27 on dist-test-slave-b83v"
server_key: "83d91038ca5df2c14cc8e66526ec82e3"
server_key_iv: "9d353ebe66b70373fc6636c9ece96bc4"
server_key_version: "encryptionkey@0"
I20260928 20:19:27.512712 3685 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:19:27.513597 3700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:27.513919 3685 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:19:27.514025 3685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/minicluster-data/master-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/minicluster-data/master-0/wal
uuid: "b8ed57f8198445929fc6676620305d9c"
format_stamp: "Formatted at 2026-09-28 20:19:27 on dist-test-slave-b83v"
server_key: "83d91038ca5df2c14cc8e66526ec82e3"
server_key_iv: "9d353ebe66b70373fc6636c9ece96bc4"
server_key_version: "encryptionkey@0"
I20260928 20:19:27.514132 3685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:19:27.532150 3685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:19:27.534323 3685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:19:27.534489 3685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:19:27.538678 3685 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.62:39501
I20260928 20:19:27.538718 3752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.62:39501 every 8 connection(s)
I20260928 20:19:27.539090 3685 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
I20260928 20:19:27.540167 3753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:27.542599 3753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8ed57f8198445929fc6676620305d9c: Bootstrap starting.
I20260928 20:19:27.543198 3753 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8ed57f8198445929fc6676620305d9c: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:27.543512 3753 log.cc:824] T 00000000000000000000000000000000 P b8ed57f8198445929fc6676620305d9c: Log is configured to *not* fsync() on all Append() calls
I20260928 20:19:27.544293 3753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8ed57f8198445929fc6676620305d9c: No bootstrap required, opened a new log
I20260928 20:19:27.546049 3753 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8ed57f8198445929fc6676620305d9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ed57f8198445929fc6676620305d9c" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 39501 } }
I20260928 20:19:27.546235 3753 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8ed57f8198445929fc6676620305d9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:27.546278 3753 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8ed57f8198445929fc6676620305d9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8ed57f8198445929fc6676620305d9c, State: Initialized, Role: FOLLOWER
I20260928 20:19:27.546387 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 3685
I20260928 20:19:27.546485 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/minicluster-data/master-0/wal/instance
I20260928 20:19:27.546473 3753 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8ed57f8198445929fc6676620305d9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ed57f8198445929fc6676620305d9c" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 39501 } }
I20260928 20:19:27.546579 3753 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8ed57f8198445929fc6676620305d9c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:19:27.546622 3753 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8ed57f8198445929fc6676620305d9c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:19:27.546644 3144 external_mini_cluster.cc:1428] Setting key a9f33a12e077d8eb66e2cc4f0cc6a8c9
I20260928 20:19:27.546682 3753 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8ed57f8198445929fc6676620305d9c [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:27.547441 3753 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8ed57f8198445929fc6676620305d9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ed57f8198445929fc6676620305d9c" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 39501 } }
I20260928 20:19:27.547573 3753 leader_election.cc:302] T 00000000000000000000000000000000 P b8ed57f8198445929fc6676620305d9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8ed57f8198445929fc6676620305d9c; no voters:
I20260928 20:19:27.547755 3753 leader_election.cc:288] T 00000000000000000000000000000000 P b8ed57f8198445929fc6676620305d9c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:19:27.547816 3758 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8ed57f8198445929fc6676620305d9c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:19:27.547957 3758 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8ed57f8198445929fc6676620305d9c [term 1 LEADER]: Becoming Leader. State: Replica: b8ed57f8198445929fc6676620305d9c, State: Running, Role: LEADER
I20260928 20:19:27.548043 3753 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8ed57f8198445929fc6676620305d9c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:19:27.548058 3758 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8ed57f8198445929fc6676620305d9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ed57f8198445929fc6676620305d9c" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 39501 } }
I20260928 20:19:27.548450 3759 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8ed57f8198445929fc6676620305d9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b8ed57f8198445929fc6676620305d9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ed57f8198445929fc6676620305d9c" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 39501 } } }
I20260928 20:19:27.548547 3759 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8ed57f8198445929fc6676620305d9c [sys.catalog]: This master's current role is: LEADER
I20260928 20:19:27.548759 3765 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:19:27.548800 3760 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8ed57f8198445929fc6676620305d9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8ed57f8198445929fc6676620305d9c. Latest consensus state: current_term: 1 leader_uuid: "b8ed57f8198445929fc6676620305d9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ed57f8198445929fc6676620305d9c" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 39501 } } }
I20260928 20:19:27.548966 3760 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8ed57f8198445929fc6676620305d9c [sys.catalog]: This master's current role is: LEADER
I20260928 20:19:27.549340 3765 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:19:27.550987 3765 catalog_manager.cc:1347] Generated new cluster ID: 9d201939ed724857b264465c0a19668d
I20260928 20:19:27.551041 3765 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 20:19:27 dist-test-slave-b83v krb5kdc[3672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:19:27.560024 3757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:27.547500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47930 (local address 127.3.18.62:39501)
0928 20:19:27.547721 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:27.547729 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:27.547818 (+ 89us) server_negotiation.cc:407] Connection header received
0928 20:19:27.547953 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:27.547961 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:27.548025 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:27.549048 (+ 1023us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:27.549163 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:27.550678 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:27.551743 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:27.552363 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:27.552457 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:27.555317 (+ 2860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:27.555322 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:27.555323 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:27.555337 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:27.558248 (+ 2911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:27.558641 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:27.558644 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:27.558645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:27.558765 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:27.558969 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:27.558970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:27.558970 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:27.559318 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:27.559476 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:27.559587 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 20:19:27.559743 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":125,"threads_started":1}
I20260928 20:19:27.572333 3765 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:19:27.572896 3765 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:19:27.577498 3765 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8ed57f8198445929fc6676620305d9c: Generated new TSK 0
I20260928 20:19:27.577698 3765 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.18.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab.
Entry for principal kudu/127.3.18.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab.
I20260928 20:19:27.611425 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.1:0
--local_ip_for_outbound_sockets=127.3.18.1
--webserver_interface=127.3.18.1
--webserver_port=0
--tserver_master_addrs=127.3.18.62:39501
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab
--principal=kudu/127.3.18.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:40409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:19:27.695246 3779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:19:27.695448 3779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:19:27.695472 3779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:19:27.696552 3779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:19:27.696594 3779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:19:27.696622 3779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.1
I20260928 20:19:27.698040 3779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:40409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab
--principal=kudu/127.3.18.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.18.1
--webserver_port=0
--tserver_master_addrs=127.3.18.62:39501
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.1
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:19:27.698230 3779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:19:27.698424 3779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:19:27.703372 3787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:27.703392 3784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:27.703604 3779 server_base.cc:1034] running on GCE node
W20260928 20:19:27.703379 3785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:27.703886 3779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:19:27.704141 3779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:19:27.705283 3779 hybrid_clock.cc:648] HybridClock initialized: now 1790626767705255 us; error 57 us; skew 500 ppm
Sep 28 20:19:27 dist-test-slave-b83v krb5kdc[3672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626767, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:19:27.707892 3779 init.cc:377] Logged in from keytab as kudu/127.3.18.1@KRBTEST.COM (short username kudu)
I20260928 20:19:27.708340 3779 webserver.cc:460] Webserver started at http://127.3.18.1:43591/ using document root <none> and password file <none>
I20260928 20:19:27.708587 3779 fs_manager.cc:359] Metadata directory not provided
I20260928 20:19:27.708655 3779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:19:27.708801 3779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:19:27.709924 3779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/minicluster-data/ts-0/data/instance:
uuid: "11663b05c4ce4f25b5d654b30130a058"
format_stamp: "Formatted at 2026-09-28 20:19:27 on dist-test-slave-b83v"
server_key: "604e051abacc2fcce7d513997b4b1e26"
server_key_iv: "eeeaf54930e0fa4ebab90300c65ae090"
server_key_version: "encryptionkey@0"
I20260928 20:19:27.710336 3779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/minicluster-data/ts-0/wal/instance:
uuid: "11663b05c4ce4f25b5d654b30130a058"
format_stamp: "Formatted at 2026-09-28 20:19:27 on dist-test-slave-b83v"
server_key: "604e051abacc2fcce7d513997b4b1e26"
server_key_iv: "eeeaf54930e0fa4ebab90300c65ae090"
server_key_version: "encryptionkey@0"
I20260928 20:19:27.711860 3779 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260928 20:19:27.712741 3794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:27.712924 3779 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:19:27.713088 3779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/minicluster-data/ts-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/minicluster-data/ts-0/wal
uuid: "11663b05c4ce4f25b5d654b30130a058"
format_stamp: "Formatted at 2026-09-28 20:19:27 on dist-test-slave-b83v"
server_key: "604e051abacc2fcce7d513997b4b1e26"
server_key_iv: "eeeaf54930e0fa4ebab90300c65ae090"
server_key_version: "encryptionkey@0"
I20260928 20:19:27.713174 3779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:19:27.727300 3779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:19:27.729956 3779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:19:27.730204 3779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:19:27.730475 3779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:19:27.730808 3779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:19:27.730858 3779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:27.730908 3779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:19:27.730948 3779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:27.737162 3779 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.1:36765
I20260928 20:19:27.737238 3907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.1:36765 every 8 connection(s)
I20260928 20:19:27.737588 3779 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
Sep 28 20:19:27 dist-test-slave-b83v krb5kdc[3672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626767, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.1@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:19:27.746507 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 3779
I20260928 20:19:27.746624 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/minicluster-data/ts-0/wal/instance
I20260928 20:19:27.746819 3144 external_mini_cluster.cc:1428] Setting key 4a642f3090e605e6cdff39b35161340c
I20260928 20:19:27.749257 3757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:27.738528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:54345 (local address 127.3.18.62:39501)
0928 20:19:27.738575 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:27.738577 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:27.738787 (+ 210us) server_negotiation.cc:407] Connection header received
0928 20:19:27.739715 (+ 928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:27.739716 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:27.739732 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:27.739898 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:27.740635 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:27.741178 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:27.742602 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:27.742811 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:27.742892 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:27.745466 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:27.745467 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:27.745468 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:27.745480 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:27.747103 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:27.747738 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:27.747740 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:27.747740 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:27.747856 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:27.748255 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:27.748257 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:27.748257 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:27.748475 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:27.748559 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:27.748988 (+ 429us) server_negotiation.cc:299] Negotiation successful
0928 20:19:27.749109 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 20:19:27.749409 3910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:27.738462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:39501 (local address 127.3.18.1:54345)
0928 20:19:27.738634 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:27.738678 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:27.739614 (+ 936us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:27.740042 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:27.740046 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:27.740321 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:27.740542 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:27.740546 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:27.741300 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:27.741301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:27.742470 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:27.742473 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:27.742924 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:27.742925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:27.743041 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:27.743661 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:19:27.743670 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:27.745349 (+ 1679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:27.747217 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:27.747227 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:27.747230 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:27.747607 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:27.747967 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:27.747968 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:27.747969 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:27.748171 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:27.748566 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:27.748572 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:27.748762 (+ 190us) client_negotiation.cc:769] Sending connection context
0928 20:19:27.748868 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 20:19:27.749170 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":100,"threads_started":1}
I20260928 20:19:27.749998 3908 heartbeater.cc:347] Connected to a master server at 127.3.18.62:39501
I20260928 20:19:27.750123 3908 heartbeater.cc:464] Registering TS with master...
I20260928 20:19:27.750331 3908 heartbeater.cc:511] Master 127.3.18.62:39501 requested a full tablet report, sending...
I20260928 20:19:27.750976 3717 ts_manager.cc:194] Registered new tserver with Master: 11663b05c4ce4f25b5d654b30130a058 (127.3.18.1:36765)
I20260928 20:19:27.752058 3717 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.18.1@KRBTEST.COM'} at 127.3.18.1:54345
WARNING: no policy specified for kudu/127.3.18.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab.
Entry for principal kudu/127.3.18.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab.
I20260928 20:19:27.775966 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.2:0
--local_ip_for_outbound_sockets=127.3.18.2
--webserver_interface=127.3.18.2
--webserver_port=0
--tserver_master_addrs=127.3.18.62:39501
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab
--principal=kudu/127.3.18.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:40409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:19:27.859936 3913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:19:27.860165 3913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:19:27.860193 3913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:19:27.861424 3913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:19:27.861476 3913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:19:27.861505 3913 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.2
I20260928 20:19:27.863050 3913 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:40409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab
--principal=kudu/127.3.18.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.18.2
--webserver_port=0
--tserver_master_addrs=127.3.18.62:39501
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3913
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.2
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:19:27.863261 3913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:19:27.863456 3913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:19:27.868741 3918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:27.868757 3919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:27.868736 3921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:27.869026 3913 server_base.cc:1034] running on GCE node
I20260928 20:19:27.869318 3913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:19:27.869534 3913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:19:27.870714 3913 hybrid_clock.cc:648] HybridClock initialized: now 1790626767870687 us; error 65 us; skew 500 ppm
Sep 28 20:19:27 dist-test-slave-b83v krb5kdc[3672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626767, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:19:27.873399 3913 init.cc:377] Logged in from keytab as kudu/127.3.18.2@KRBTEST.COM (short username kudu)
I20260928 20:19:27.873926 3913 webserver.cc:460] Webserver started at http://127.3.18.2:42597/ using document root <none> and password file <none>
I20260928 20:19:27.874186 3913 fs_manager.cc:359] Metadata directory not provided
I20260928 20:19:27.874253 3913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:19:27.874395 3913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:19:27.875458 3913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/minicluster-data/ts-1/data/instance:
uuid: "b789a13c3d304e32896d16d0b72de8da"
format_stamp: "Formatted at 2026-09-28 20:19:27 on dist-test-slave-b83v"
server_key: "8e6447df38f7c386d7938e30ef508c10"
server_key_iv: "1174f306ea3aedbbb7c2d7c35633614b"
server_key_version: "encryptionkey@0"
I20260928 20:19:27.875880 3913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/minicluster-data/ts-1/wal/instance:
uuid: "b789a13c3d304e32896d16d0b72de8da"
format_stamp: "Formatted at 2026-09-28 20:19:27 on dist-test-slave-b83v"
server_key: "8e6447df38f7c386d7938e30ef508c10"
server_key_iv: "1174f306ea3aedbbb7c2d7c35633614b"
server_key_version: "encryptionkey@0"
I20260928 20:19:27.877301 3913 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:19:27.878248 3928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:27.878450 3913 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:19:27.878561 3913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/minicluster-data/ts-1/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/minicluster-data/ts-1/wal
uuid: "b789a13c3d304e32896d16d0b72de8da"
format_stamp: "Formatted at 2026-09-28 20:19:27 on dist-test-slave-b83v"
server_key: "8e6447df38f7c386d7938e30ef508c10"
server_key_iv: "1174f306ea3aedbbb7c2d7c35633614b"
server_key_version: "encryptionkey@0"
I20260928 20:19:27.878647 3913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:19:27.890986 3913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:19:27.893399 3913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:19:27.893571 3913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:19:27.893827 3913 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:19:27.894160 3913 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:19:27.894220 3913 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:27.894263 3913 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:19:27.894304 3913 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:27.900187 3913 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.2:37725
I20260928 20:19:27.900256 4041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.2:37725 every 8 connection(s)
I20260928 20:19:27.900588 3913 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
I20260928 20:19:27.901058 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 3913
I20260928 20:19:27.901160 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/minicluster-data/ts-1/wal/instance
I20260928 20:19:27.901328 3144 external_mini_cluster.cc:1428] Setting key a44e6df512dde9acfdb9a41ac57aa63a
Sep 28 20:19:27 dist-test-slave-b83v krb5kdc[3672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626767, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.2@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.18.3@KRBTEST.COM; defaulting to no policy
I20260928 20:19:27.914994 3757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:27.902135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:41507 (local address 127.3.18.62:39501)
0928 20:19:27.902190 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:27.902193 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:27.902410 (+ 217us) server_negotiation.cc:407] Connection header received
0928 20:19:27.903626 (+ 1216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:27.903627 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:27.903637 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:27.903683 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:27.904673 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:27.905248 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:27.908385 (+ 3137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:27.908622 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:27.908696 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:27.911500 (+ 2804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:27.911502 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:27.911503 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:27.911511 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:27.913226 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:27.913792 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:27.913793 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:27.913793 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:27.913856 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:27.914192 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:27.914192 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:27.914193 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:27.914393 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:27.914461 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:27.914758 (+ 297us) server_negotiation.cc:299] Negotiation successful
0928 20:19:27.914863 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.3@KRBTEST.COM" created.
I20260928 20:19:27.915246 4044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:27.902021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:39501 (local address 127.3.18.2:41507)
0928 20:19:27.902282 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:27.902332 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:27.903485 (+ 1153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:27.903798 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:27.903803 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:27.904234 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:27.904558 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:27.904565 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:27.905355 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:27.905357 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:27.908249 (+ 2892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:27.908254 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:27.908721 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:27.908723 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:27.908902 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:27.909552 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:19:27.909565 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:27.911361 (+ 1796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:27.913356 (+ 1995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:27.913365 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:27.913368 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:27.913694 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:27.913948 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:27.913949 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:27.913950 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:27.914099 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:27.914482 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:27.914488 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:27.914633 (+ 145us) client_negotiation.cc:769] Sending connection context
0928 20:19:27.914729 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 20:19:27.914877 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":176,"threads_started":1}
I20260928 20:19:27.915925 4042 heartbeater.cc:347] Connected to a master server at 127.3.18.62:39501
I20260928 20:19:27.916043 4042 heartbeater.cc:464] Registering TS with master...
I20260928 20:19:27.916273 4042 heartbeater.cc:511] Master 127.3.18.62:39501 requested a full tablet report, sending...
I20260928 20:19:27.916703 3717 ts_manager.cc:194] Registered new tserver with Master: b789a13c3d304e32896d16d0b72de8da (127.3.18.2:37725)
I20260928 20:19:27.917132 3717 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.18.2@KRBTEST.COM'} at 127.3.18.2:41507
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab.
Entry for principal kudu/127.3.18.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab.
I20260928 20:19:27.933071 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.3:0
--local_ip_for_outbound_sockets=127.3.18.3
--webserver_interface=127.3.18.3
--webserver_port=0
--tserver_master_addrs=127.3.18.62:39501
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab
--principal=kudu/127.3.18.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:40409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:19:28.015630 4047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:19:28.016003 4047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:19:28.016176 4047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:19:28.017696 4047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:19:28.017755 4047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:19:28.017800 4047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.3
I20260928 20:19:28.019289 4047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:40409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab
--principal=kudu/127.3.18.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.18.3
--webserver_port=0
--tserver_master_addrs=127.3.18.62:39501
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.3
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:19:28.019562 4047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:19:28.019861 4047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:19:28.024868 4053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:28.024931 4052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:28.024986 4047 server_base.cc:1034] running on GCE node
W20260928 20:19:28.025126 4055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:28.025370 4047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:19:28.025636 4047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:19:28.026799 4047 hybrid_clock.cc:648] HybridClock initialized: now 1790626768026784 us; error 29 us; skew 500 ppm
Sep 28 20:19:28 dist-test-slave-b83v krb5kdc[3672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626768, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:19:28.029422 4047 init.cc:377] Logged in from keytab as kudu/127.3.18.3@KRBTEST.COM (short username kudu)
I20260928 20:19:28.029881 4047 webserver.cc:460] Webserver started at http://127.3.18.3:35793/ using document root <none> and password file <none>
I20260928 20:19:28.030256 4047 fs_manager.cc:359] Metadata directory not provided
I20260928 20:19:28.030341 4047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:19:28.030483 4047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:19:28.031685 4047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/minicluster-data/ts-2/data/instance:
uuid: "4f90870de3794eadb3973cea728aad12"
format_stamp: "Formatted at 2026-09-28 20:19:28 on dist-test-slave-b83v"
server_key: "1b4f8d6271e96819995c7755462fb7e9"
server_key_iv: "117b22fe529d4a077ed56efc63c98e07"
server_key_version: "encryptionkey@0"
I20260928 20:19:28.032150 4047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/minicluster-data/ts-2/wal/instance:
uuid: "4f90870de3794eadb3973cea728aad12"
format_stamp: "Formatted at 2026-09-28 20:19:28 on dist-test-slave-b83v"
server_key: "1b4f8d6271e96819995c7755462fb7e9"
server_key_iv: "117b22fe529d4a077ed56efc63c98e07"
server_key_version: "encryptionkey@0"
I20260928 20:19:28.033533 4047 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:19:28.034469 4062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:28.034710 4047 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:19:28.034822 4047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/minicluster-data/ts-2/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/minicluster-data/ts-2/wal
uuid: "4f90870de3794eadb3973cea728aad12"
format_stamp: "Formatted at 2026-09-28 20:19:28 on dist-test-slave-b83v"
server_key: "1b4f8d6271e96819995c7755462fb7e9"
server_key_iv: "117b22fe529d4a077ed56efc63c98e07"
server_key_version: "encryptionkey@0"
I20260928 20:19:28.034914 4047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:19:28.045275 4047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:19:28.047683 4047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:19:28.047871 4047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:19:28.048204 4047 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:19:28.048583 4047 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:19:28.048635 4047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:28.048676 4047 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:19:28.048693 4047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:28.054764 4047 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.3:36051
I20260928 20:19:28.054804 4175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.3:36051 every 8 connection(s)
I20260928 20:19:28.055128 4047 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
I20260928 20:19:28.058401 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 4047
I20260928 20:19:28.058545 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790626764250411-3144-0/minicluster-data/ts-2/wal/instance
I20260928 20:19:28.058744 3144 external_mini_cluster.cc:1428] Setting key 3165a7485bc34233b3765d7f6c059dc3
Sep 28 20:19:28 dist-test-slave-b83v krb5kdc[3672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626768, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.3@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:19:28.065289 3757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:28.055902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:60915 (local address 127.3.18.62:39501)
0928 20:19:28.055975 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:28.055977 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:28.056263 (+ 286us) server_negotiation.cc:407] Connection header received
0928 20:19:28.057243 (+ 980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:28.057245 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:28.057256 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:28.057323 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:28.057918 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:28.058412 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:28.060035 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:28.060241 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:28.060327 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:28.062617 (+ 2290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:28.062619 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:28.062619 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:28.062627 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:28.063902 (+ 1275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:28.064327 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:28.064329 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:28.064329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:28.064391 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:28.064644 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:28.064645 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:28.064646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:28.064764 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:28.064840 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:28.065105 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 20:19:28.065195 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 20:19:28.065490 4178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:28.055887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:39501 (local address 127.3.18.3:60915)
0928 20:19:28.056109 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:28.056151 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:28.057114 (+ 963us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:28.057381 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:28.057384 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:28.057646 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:28.057839 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:28.057843 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:28.058585 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:28.058587 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:28.059934 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:28.059936 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:28.060335 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:28.060336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:28.060475 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:28.061034 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:19:28.061043 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:28.062447 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:28.063995 (+ 1548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:28.064002 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:28.064004 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:28.064237 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:28.064458 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:28.064458 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:28.064459 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:28.064575 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:28.064829 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:28.064835 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:28.064975 (+ 140us) client_negotiation.cc:769] Sending connection context
0928 20:19:28.065085 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 20:19:28.065245 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":94,"threads_started":1}
I20260928 20:19:28.065969 4176 heartbeater.cc:347] Connected to a master server at 127.3.18.62:39501
I20260928 20:19:28.066069 4176 heartbeater.cc:464] Registering TS with master...
I20260928 20:19:28.066270 4176 heartbeater.cc:511] Master 127.3.18.62:39501 requested a full tablet report, sending...
I20260928 20:19:28.066641 3717 ts_manager.cc:194] Registered new tserver with Master: 4f90870de3794eadb3973cea728aad12 (127.3.18.3:36051)
I20260928 20:19:28.066983 3717 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.18.3@KRBTEST.COM'} at 127.3.18.3:60915
I20260928 20:19:28.070571 3144 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 20:19:28.078176 3757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:28.072035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47936 (local address 127.3.18.62:39501)
0928 20:19:28.072157 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:28.072159 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:28.072308 (+ 149us) server_negotiation.cc:407] Connection header received
0928 20:19:28.072331 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:28.072331 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:28.072341 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:28.072385 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:28.072904 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:28.073422 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:28.074088 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:28.074293 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:28.074345 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:28.075472 (+ 1127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:28.075474 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:28.075474 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:28.075480 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:28.076709 (+ 1229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:28.077209 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:28.077210 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:28.077210 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:28.077277 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:28.077564 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:28.077565 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:28.077566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:28.077735 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:28.077847 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:28.078000 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 20:19:28.078082 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 20:19:28.080555 3717 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47936:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 20:19:28.082275 3717 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 20:19:28.089654 4189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:28.085256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55504 (local address 127.3.18.3:36051)
0928 20:19:28.085539 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:28.085543 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:28.085554 (+ 11us) server_negotiation.cc:407] Connection header received
0928 20:19:28.085595 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:28.085597 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:28.085617 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:28.085696 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:19:28.085964 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:28.087219 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:28.088452 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:28.089231 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:28.089313 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:28.089374 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:28.089552 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 20:19:28.089589 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":93,"threads_started":1}
I20260928 20:19:28.089648 4188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:28.085131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.3:36051 (local address 127.0.0.1:55504)
0928 20:19:28.085338 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:28.085382 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:28.085453 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:28.085797 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:28.085801 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:28.085803 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:19:28.085878 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:28.085880 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:28.087331 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:28.087333 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:28.088358 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:28.088361 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:28.089332 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:28.089334 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:28.089439 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:28.089456 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 20:19:28.089532 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 20:19:28.089578 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":29,"thread_start_us":89,"threads_started":1}
I20260928 20:19:28.090045 4187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:28.085364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.1:36765 (local address 127.0.0.1:48636)
0928 20:19:28.085656 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:28.085667 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:28.085718 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:28.086140 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:28.086141 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:28.086143 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:19:28.086191 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:28.086193 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:28.087100 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:28.087102 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:28.088462 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:28.088465 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:28.089806 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:28.089807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:28.089870 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:28.089876 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 20:19:28.089935 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 20:19:28.089981 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":139,"threads_started":1}
I20260928 20:19:28.090377 4191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:28.085131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.2:37725 (local address 127.0.0.1:41492)
0928 20:19:28.085557 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:28.085606 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:28.085670 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:28.085987 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:28.085987 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:28.085989 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:19:28.086023 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:28.086025 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:28.087289 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:28.087290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:28.088637 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:28.088639 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:28.090151 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:28.090152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:28.090210 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:28.090214 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:19:28.090273 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 20:19:28.090309 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"mutex_wait_us":30,"thread_start_us":81,"threads_started":1}
I20260928 20:19:28.090687 4190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:28.085276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41492 (local address 127.3.18.2:37725)
0928 20:19:28.085449 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:28.085454 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:28.085735 (+ 281us) server_negotiation.cc:407] Connection header received
0928 20:19:28.085781 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:28.085783 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:28.085826 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:28.085901 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:19:28.086097 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:28.087167 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:28.088720 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:28.089239 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:28.089342 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:28.089403 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:28.090557 (+ 1154us) server_negotiation.cc:299] Negotiation successful
0928 20:19:28.090607 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":70,"threads_started":1}
I20260928 20:19:28.090546 4110 tablet_service.cc:1467] Processing CreateTablet for tablet 0a8c4e0397de472fac73c73a94390842 (DEFAULT_TABLE table=test-table [id=79d4a9cb8d6d4d49aabdb88ad48631df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:19:28.090970 4110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a8c4e0397de472fac73c73a94390842. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:28.091379 3976 tablet_service.cc:1467] Processing CreateTablet for tablet 0a8c4e0397de472fac73c73a94390842 (DEFAULT_TABLE table=test-table [id=79d4a9cb8d6d4d49aabdb88ad48631df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:19:28.091694 3976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a8c4e0397de472fac73c73a94390842. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:28.094004 4193 tablet_bootstrap.cc:492] T 0a8c4e0397de472fac73c73a94390842 P 4f90870de3794eadb3973cea728aad12: Bootstrap starting.
I20260928 20:19:28.094166 4192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:28.085655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48636 (local address 127.3.18.1:36765)
0928 20:19:28.085874 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:28.085878 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:28.085889 (+ 11us) server_negotiation.cc:407] Connection header received
0928 20:19:28.085943 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:28.085945 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:28.085963 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:28.086044 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:19:28.086282 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:28.086987 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:28.088568 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:28.089239 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:28.089313 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:28.089374 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:28.089958 (+ 584us) server_negotiation.cc:299] Negotiation successful
0928 20:19:28.094063 (+ 4105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":106,"threads_started":1}
I20260928 20:19:28.094867 4193 tablet_bootstrap.cc:654] T 0a8c4e0397de472fac73c73a94390842 P 4f90870de3794eadb3973cea728aad12: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:28.094882 3842 tablet_service.cc:1467] Processing CreateTablet for tablet 0a8c4e0397de472fac73c73a94390842 (DEFAULT_TABLE table=test-table [id=79d4a9cb8d6d4d49aabdb88ad48631df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:19:28.095237 3842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a8c4e0397de472fac73c73a94390842. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:28.095432 4193 log.cc:824] T 0a8c4e0397de472fac73c73a94390842 P 4f90870de3794eadb3973cea728aad12: Log is configured to *not* fsync() on all Append() calls
I20260928 20:19:28.095939 4194 tablet_bootstrap.cc:492] T 0a8c4e0397de472fac73c73a94390842 P b789a13c3d304e32896d16d0b72de8da: Bootstrap starting.
I20260928 20:19:28.096236 4193 tablet_bootstrap.cc:492] T 0a8c4e0397de472fac73c73a94390842 P 4f90870de3794eadb3973cea728aad12: No bootstrap required, opened a new log
I20260928 20:19:28.096320 4193 ts_tablet_manager.cc:1397] T 0a8c4e0397de472fac73c73a94390842 P 4f90870de3794eadb3973cea728aad12: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 20:19:28.096609 4194 tablet_bootstrap.cc:654] T 0a8c4e0397de472fac73c73a94390842 P b789a13c3d304e32896d16d0b72de8da: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:28.097214 4194 log.cc:824] T 0a8c4e0397de472fac73c73a94390842 P b789a13c3d304e32896d16d0b72de8da: Log is configured to *not* fsync() on all Append() calls
I20260928 20:19:28.098022 4194 tablet_bootstrap.cc:492] T 0a8c4e0397de472fac73c73a94390842 P b789a13c3d304e32896d16d0b72de8da: No bootstrap required, opened a new log
I20260928 20:19:28.098109 4194 ts_tablet_manager.cc:1397] T 0a8c4e0397de472fac73c73a94390842 P b789a13c3d304e32896d16d0b72de8da: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 20:19:28.098222 4193 raft_consensus.cc:348] T 0a8c4e0397de472fac73c73a94390842 P 4f90870de3794eadb3973cea728aad12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f90870de3794eadb3973cea728aad12" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36051 } } peers { permanent_uuid: "b789a13c3d304e32896d16d0b72de8da" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 37725 } } peers { permanent_uuid: "11663b05c4ce4f25b5d654b30130a058" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 36765 } }
I20260928 20:19:28.098392 4193 raft_consensus.cc:374] T 0a8c4e0397de472fac73c73a94390842 P 4f90870de3794eadb3973cea728aad12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:28.098435 4193 raft_consensus.cc:729] T 0a8c4e0397de472fac73c73a94390842 P 4f90870de3794eadb3973cea728aad12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f90870de3794eadb3973cea728aad12, State: Initialized, Role: FOLLOWER
I20260928 20:19:28.098551 4193 consensus_queue.cc:260] T 0a8c4e0397de472fac73c73a94390842 P 4f90870de3794eadb3973cea728aad12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f90870de3794eadb3973cea728aad12" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36051 } } peers { permanent_uuid: "b789a13c3d304e32896d16d0b72de8da" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 37725 } } peers { permanent_uuid: "11663b05c4ce4f25b5d654b30130a058" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 36765 } }
I20260928 20:19:28.098865 4193 ts_tablet_manager.cc:1428] T 0a8c4e0397de472fac73c73a94390842 P 4f90870de3794eadb3973cea728aad12: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 20:19:28.098871 4197 tablet_bootstrap.cc:492] T 0a8c4e0397de472fac73c73a94390842 P 11663b05c4ce4f25b5d654b30130a058: Bootstrap starting.
I20260928 20:19:28.099020 4176 heartbeater.cc:503] Master 127.3.18.62:39501 was elected leader, sending a full tablet report...
I20260928 20:19:28.099594 4197 tablet_bootstrap.cc:654] T 0a8c4e0397de472fac73c73a94390842 P 11663b05c4ce4f25b5d654b30130a058: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:28.100216 4197 log.cc:824] T 0a8c4e0397de472fac73c73a94390842 P 11663b05c4ce4f25b5d654b30130a058: Log is configured to *not* fsync() on all Append() calls
I20260928 20:19:28.100641 4194 raft_consensus.cc:348] T 0a8c4e0397de472fac73c73a94390842 P b789a13c3d304e32896d16d0b72de8da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f90870de3794eadb3973cea728aad12" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36051 } } peers { permanent_uuid: "b789a13c3d304e32896d16d0b72de8da" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 37725 } } peers { permanent_uuid: "11663b05c4ce4f25b5d654b30130a058" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 36765 } }
I20260928 20:19:28.100835 4194 raft_consensus.cc:374] T 0a8c4e0397de472fac73c73a94390842 P b789a13c3d304e32896d16d0b72de8da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:28.100863 4194 raft_consensus.cc:729] T 0a8c4e0397de472fac73c73a94390842 P b789a13c3d304e32896d16d0b72de8da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b789a13c3d304e32896d16d0b72de8da, State: Initialized, Role: FOLLOWER
I20260928 20:19:28.101016 4194 consensus_queue.cc:260] T 0a8c4e0397de472fac73c73a94390842 P b789a13c3d304e32896d16d0b72de8da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f90870de3794eadb3973cea728aad12" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36051 } } peers { permanent_uuid: "b789a13c3d304e32896d16d0b72de8da" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 37725 } } peers { permanent_uuid: "11663b05c4ce4f25b5d654b30130a058" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 36765 } }
I20260928 20:19:28.101226 4197 tablet_bootstrap.cc:492] T 0a8c4e0397de472fac73c73a94390842 P 11663b05c4ce4f25b5d654b30130a058: No bootstrap required, opened a new log
I20260928 20:19:28.101302 4197 ts_tablet_manager.cc:1397] T 0a8c4e0397de472fac73c73a94390842 P 11663b05c4ce4f25b5d654b30130a058: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 20:19:28.101330 4194 ts_tablet_manager.cc:1428] T 0a8c4e0397de472fac73c73a94390842 P b789a13c3d304e32896d16d0b72de8da: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 20:19:28.101373 4042 heartbeater.cc:503] Master 127.3.18.62:39501 was elected leader, sending a full tablet report...
I20260928 20:19:28.102702 4197 raft_consensus.cc:348] T 0a8c4e0397de472fac73c73a94390842 P 11663b05c4ce4f25b5d654b30130a058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f90870de3794eadb3973cea728aad12" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36051 } } peers { permanent_uuid: "b789a13c3d304e32896d16d0b72de8da" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 37725 } } peers { permanent_uuid: "11663b05c4ce4f25b5d654b30130a058" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 36765 } }
I20260928 20:19:28.102833 4197 raft_consensus.cc:374] T 0a8c4e0397de472fac73c73a94390842 P 11663b05c4ce4f25b5d654b30130a058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:28.102854 4197 raft_consensus.cc:729] T 0a8c4e0397de472fac73c73a94390842 P 11663b05c4ce4f25b5d654b30130a058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11663b05c4ce4f25b5d654b30130a058, State: Initialized, Role: FOLLOWER
I20260928 20:19:28.102986 4197 consensus_queue.cc:260] T 0a8c4e0397de472fac73c73a94390842 P 11663b05c4ce4f25b5d654b30130a058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f90870de3794eadb3973cea728aad12" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36051 } } peers { permanent_uuid: "b789a13c3d304e32896d16d0b72de8da" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 37725 } } peers { permanent_uuid: "11663b05c4ce4f25b5d654b30130a058" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 36765 } }
I20260928 20:19:28.103257 4197 ts_tablet_manager.cc:1428] T 0a8c4e0397de472fac73c73a94390842 P 11663b05c4ce4f25b5d654b30130a058: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 20:19:28.103281 3908 heartbeater.cc:503] Master 127.3.18.62:39501 was elected leader, sending a full tablet report...
W20260928 20:19:28.151338 4043 tablet.cc:2369] T 0a8c4e0397de472fac73c73a94390842 P b789a13c3d304e32896d16d0b72de8da: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 20:19:28.238365 3909 tablet.cc:2369] T 0a8c4e0397de472fac73c73a94390842 P 11663b05c4ce4f25b5d654b30130a058: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 20:19:28.305712 4177 tablet.cc:2369] T 0a8c4e0397de472fac73c73a94390842 P 4f90870de3794eadb3973cea728aad12: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 20:19:28.342129 4201 raft_consensus.cc:482] T 0a8c4e0397de472fac73c73a94390842 P 11663b05c4ce4f25b5d654b30130a058 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:19:28.342264 4201 raft_consensus.cc:504] T 0a8c4e0397de472fac73c73a94390842 P 11663b05c4ce4f25b5d654b30130a058 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f90870de3794eadb3973cea728aad12" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36051 } } peers { permanent_uuid: "b789a13c3d304e32896d16d0b72de8da" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 37725 } } peers { permanent_uuid: "11663b05c4ce4f25b5d654b30130a058" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 36765 } }
I20260928 20:19:28.342684 4201 leader_election.cc:288] T 0a8c4e0397de472fac73c73a94390842 P 11663b05c4ce4f25b5d654b30130a058 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f90870de3794eadb3973cea728aad12 (127.3.18.3:36051), b789a13c3d304e32896d16d0b72de8da (127.3.18.2:37725)
I20260928 20:19:28.345907 4190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:28.343013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:41331 (local address 127.3.18.2:37725)
0928 20:19:28.343056 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:28.343059 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:28.343100 (+ 41us) server_negotiation.cc:407] Connection header received
0928 20:19:28.343195 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:28.343196 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:28.343209 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:28.343269 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:19:28.343455 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:28.344005 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:28.344986 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:28.345521 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:28.345599 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:28.345648 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:28.345779 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 20:19:28.345836 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 20:19:28.345960 4203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:28.342887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.2:37725 (local address 127.3.18.1:41331)
0928 20:19:28.343035 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:28.343047 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:28.343109 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:28.343285 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:28.343286 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:28.343289 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:19:28.343374 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:28.343377 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:28.344080 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:28.344082 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:28.344900 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:28.344902 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:28.345627 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:28.345628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:28.345704 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:28.345709 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 20:19:28.345771 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 20:19:28.345809 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":77,"threads_started":1}
I20260928 20:19:28.346053 4202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:28.342890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.3:36051 (local address 127.3.18.1:53225)
0928 20:19:28.343041 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:28.343053 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:28.343129 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:28.343346 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:28.343353 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:28.343355 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:19:28.343451 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:28.343453 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:28.344211 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:28.344212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:28.344994 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:28.344997 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:28.345690 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:28.345692 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:28.345779 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:28.345785 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 20:19:28.345844 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 20:19:28.345883 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":82,"threads_started":1}
I20260928 20:19:28.346051 4189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:28.342976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:53225 (local address 127.3.18.3:36051)
0928 20:19:28.343027 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:28.343029 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:28.343105 (+ 76us) server_negotiation.cc:407] Connection header received
0928 20:19:28.343237 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:28.343238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:28.343268 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:28.343327 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:19:28.343536 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:28.344079 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:28.345152 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:28.345535 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:28.345654 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:28.345665 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:28.345867 (+ 202us) server_negotiation.cc:299] Negotiation successful
0928 20:19:28.345915 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 20:19:28.346408 4130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a8c4e0397de472fac73c73a94390842" candidate_uuid: "11663b05c4ce4f25b5d654b30130a058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f90870de3794eadb3973cea728aad12" is_pre_election: true
I20260928 20:19:28.346467 3996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a8c4e0397de472fac73c73a94390842" candidate_uuid: "11663b05c4ce4f25b5d654b30130a058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b789a13c3d304e32896d16d0b72de8da" is_pre_election: true
I20260928 20:19:28.346580 4130 raft_consensus.cc:2454] T 0a8c4e0397de472fac73c73a94390842 P 4f90870de3794eadb3973cea728aad12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11663b05c4ce4f25b5d654b30130a058 in term 0.
I20260928 20:19:28.346683 3996 raft_consensus.cc:2454] T 0a8c4e0397de472fac73c73a94390842 P b789a13c3d304e32896d16d0b72de8da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11663b05c4ce4f25b5d654b30130a058 in term 0.
I20260928 20:19:28.346863 3796 leader_election.cc:302] T 0a8c4e0397de472fac73c73a94390842 P 11663b05c4ce4f25b5d654b30130a058 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11663b05c4ce4f25b5d654b30130a058, 4f90870de3794eadb3973cea728aad12; no voters:
I20260928 20:19:28.347004 4201 raft_consensus.cc:2781] T 0a8c4e0397de472fac73c73a94390842 P 11663b05c4ce4f25b5d654b30130a058 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 20:19:28.347101 4201 raft_consensus.cc:482] T 0a8c4e0397de472fac73c73a94390842 P 11663b05c4ce4f25b5d654b30130a058 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 20:19:28.347137 4201 raft_consensus.cc:3037] T 0a8c4e0397de472fac73c73a94390842 P 11663b05c4ce4f25b5d654b30130a058 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:28.347965 4201 raft_consensus.cc:504] T 0a8c4e0397de472fac73c73a94390842 P 11663b05c4ce4f25b5d654b30130a058 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f90870de3794eadb3973cea728aad12" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36051 } } peers { permanent_uuid: "b789a13c3d304e32896d16d0b72de8da" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 37725 } } peers { permanent_uuid: "11663b05c4ce4f25b5d654b30130a058" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 36765 } }
I20260928 20:19:28.348136 4201 leader_election.cc:288] T 0a8c4e0397de472fac73c73a94390842 P 11663b05c4ce4f25b5d654b30130a058 [CANDIDATE]: Term 1 election: Requested vote from peers 4f90870de3794eadb3973cea728aad12 (127.3.18.3:36051), b789a13c3d304e32896d16d0b72de8da (127.3.18.2:37725)
I20260928 20:19:28.348310 4130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a8c4e0397de472fac73c73a94390842" candidate_uuid: "11663b05c4ce4f25b5d654b30130a058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f90870de3794eadb3973cea728aad12"
I20260928 20:19:28.348331 3996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a8c4e0397de472fac73c73a94390842" candidate_uuid: "11663b05c4ce4f25b5d654b30130a058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b789a13c3d304e32896d16d0b72de8da"
I20260928 20:19:28.348414 3996 raft_consensus.cc:3037] T 0a8c4e0397de472fac73c73a94390842 P b789a13c3d304e32896d16d0b72de8da [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:28.348400 4130 raft_consensus.cc:3037] T 0a8c4e0397de472fac73c73a94390842 P 4f90870de3794eadb3973cea728aad12 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:28.349262 3996 raft_consensus.cc:2454] T 0a8c4e0397de472fac73c73a94390842 P b789a13c3d304e32896d16d0b72de8da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11663b05c4ce4f25b5d654b30130a058 in term 1.
I20260928 20:19:28.349432 4130 raft_consensus.cc:2454] T 0a8c4e0397de472fac73c73a94390842 P 4f90870de3794eadb3973cea728aad12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11663b05c4ce4f25b5d654b30130a058 in term 1.
I20260928 20:19:28.349484 3795 leader_election.cc:302] T 0a8c4e0397de472fac73c73a94390842 P 11663b05c4ce4f25b5d654b30130a058 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11663b05c4ce4f25b5d654b30130a058, b789a13c3d304e32896d16d0b72de8da; no voters:
I20260928 20:19:28.349653 4201 raft_consensus.cc:2781] T 0a8c4e0397de472fac73c73a94390842 P 11663b05c4ce4f25b5d654b30130a058 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:19:28.349817 4201 raft_consensus.cc:686] T 0a8c4e0397de472fac73c73a94390842 P 11663b05c4ce4f25b5d654b30130a058 [term 1 LEADER]: Becoming Leader. State: Replica: 11663b05c4ce4f25b5d654b30130a058, State: Running, Role: LEADER
I20260928 20:19:28.349888 4201 consensus_queue.cc:237] T 0a8c4e0397de472fac73c73a94390842 P 11663b05c4ce4f25b5d654b30130a058 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f90870de3794eadb3973cea728aad12" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36051 } } peers { permanent_uuid: "b789a13c3d304e32896d16d0b72de8da" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 37725 } } peers { permanent_uuid: "11663b05c4ce4f25b5d654b30130a058" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 36765 } }
I20260928 20:19:28.350879 3717 catalog_manager.cc:5495] T 0a8c4e0397de472fac73c73a94390842 P 11663b05c4ce4f25b5d654b30130a058 reported cstate change: term changed from 0 to 1, leader changed from <none> to 11663b05c4ce4f25b5d654b30130a058 (127.3.18.1). New cstate: current_term: 1 leader_uuid: "11663b05c4ce4f25b5d654b30130a058" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f90870de3794eadb3973cea728aad12" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b789a13c3d304e32896d16d0b72de8da" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 37725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11663b05c4ce4f25b5d654b30130a058" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 36765 } health_report { overall_health: HEALTHY } } }
Sep 28 20:19:28 dist-test-slave-b83v krb5kdc[3672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.18.1@KRBTEST.COM
I20260928 20:19:28.379722 4192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:28.370678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48650 (local address 127.3.18.1:36765)
0928 20:19:28.370756 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:28.370759 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:28.370868 (+ 109us) server_negotiation.cc:407] Connection header received
0928 20:19:28.370957 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:28.370958 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:28.370975 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:28.371037 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:28.371664 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:28.372220 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:28.373067 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:28.373267 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:28.373375 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:28.376032 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:28.376047 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:28.376047 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:28.376056 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:28.378235 (+ 2179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:28.378634 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:28.378639 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:28.378639 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:28.378709 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:28.379046 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:28.379047 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:28.379047 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:28.379235 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:28.379378 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:28.379564 (+ 186us) server_negotiation.cc:299] Negotiation successful
0928 20:19:28.379648 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Sep 28 20:19:28 dist-test-slave-b83v krb5kdc[3672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626768, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 20:19:28 dist-test-slave-b83v krb5kdc[3672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626768, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.18.1@KRBTEST.COM
I20260928 20:19:28.405187 4192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:28.397450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48658 (local address 127.3.18.1:36765)
0928 20:19:28.397544 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:28.397546 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:28.397675 (+ 129us) server_negotiation.cc:407] Connection header received
0928 20:19:28.397796 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:28.397797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:28.397808 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:28.397884 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:28.398488 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:28.398985 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:28.399785 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:28.399995 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:28.400080 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:28.402391 (+ 2311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:28.402393 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:28.402394 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:28.402402 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:28.403951 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:28.404319 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:28.404320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:28.404321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:28.404381 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:28.404580 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:28.404581 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:28.404581 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:28.404685 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:28.404803 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:28.404987 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 20:19:28.405070 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Sep 28 20:19:28 dist-test-slave-b83v krb5kdc[3672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 20:19:28 dist-test-slave-b83v krb5kdc[3672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.18.1@KRBTEST.COM
I20260928 20:19:28.428419 4192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:28.420692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48660 (local address 127.3.18.1:36765)
0928 20:19:28.420774 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:28.420776 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:28.420828 (+ 52us) server_negotiation.cc:407] Connection header received
0928 20:19:28.420895 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:28.420896 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:28.420905 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:28.420946 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:28.421479 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:28.422017 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:28.422856 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:28.423053 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:28.423117 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:28.425989 (+ 2872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:28.425990 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:28.425991 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:28.425997 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:28.427346 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:28.427678 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:28.427679 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:28.427680 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:28.427732 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:28.427986 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:28.427987 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:28.427987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:28.428083 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:28.428207 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:28.428255 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 20:19:28.428314 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260928 20:19:28.428907 3822 tablet_service.cc:2908] Rejecting scan request for tablet 0a8c4e0397de472fac73c73a94390842: Uninitialized: clean time has not yet been initialized
I20260928 20:19:28.429793 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 3779
I20260928 20:19:28.434859 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 3913
I20260928 20:19:28.439359 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 4047
I20260928 20:19:28.443878 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 3685
2026-09-28T20:19:28Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2211 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:19:28 dist-test-slave-b83v krb5kdc[4218](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:19:28 dist-test-slave-b83v krb5kdc[4218](info): set up 2 sockets
Sep 28 20:19:28 dist-test-slave-b83v krb5kdc[4218](info): commencing operation
krb5kdc: starting...
W20260928 20:19:29.503849 3144 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Sep 28 20:19:29 dist-test-slave-b83v krb5kdc[4218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:19:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:19:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.18.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab.
Entry for principal kudu/127.3.18.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab.
I20260928 20:19:29.628750 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:40329
--webserver_interface=127.3.18.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:45961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.18.62:40329
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:19:29.717988 4231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:19:29.718252 4231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:19:29.718291 4231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:19:29.719465 4231 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:19:29.719532 4231 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 20:19:29.719558 4231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:19:29.719594 4231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:19:29.719641 4231 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:19:29.719676 4231 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:19:29.721338 4231 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:45961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.18.62:40329
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:40329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.18.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4231
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:19:29.721632 4231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:19:29.721868 4231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:19:29.727046 4237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:29.727068 4236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:29.727131 4231 server_base.cc:1034] running on GCE node
W20260928 20:19:29.727061 4239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:29.727408 4231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:19:29.727701 4231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:19:29.728886 4231 hybrid_clock.cc:648] HybridClock initialized: now 1790626769728877 us; error 35 us; skew 500 ppm
Sep 28 20:19:29 dist-test-slave-b83v krb5kdc[4218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626769, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:19:29.731501 4231 init.cc:377] Logged in from keytab as kudu/127.3.18.62@KRBTEST.COM (short username kudu)
I20260928 20:19:29.732031 4231 webserver.cc:460] Webserver started at http://127.3.18.62:40363/ using document root <none> and password file <none>
I20260928 20:19:29.732231 4231 fs_manager.cc:359] Metadata directory not provided
I20260928 20:19:29.732275 4231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:19:29.732366 4231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:19:29.733417 4231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/master-0/data/instance:
uuid: "3b2eb202db314d8e8d097278b55d724d"
format_stamp: "Formatted at 2026-09-28 20:19:29 on dist-test-slave-b83v"
server_key: "733bc8f92d5e2a2e99bb338d5e41a443"
server_key_iv: "d619f1e04ea40ab7b27c46e6b3ecda0a"
server_key_version: "encryptionkey@0"
I20260928 20:19:29.733754 4231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/master-0/wal/instance:
uuid: "3b2eb202db314d8e8d097278b55d724d"
format_stamp: "Formatted at 2026-09-28 20:19:29 on dist-test-slave-b83v"
server_key: "733bc8f92d5e2a2e99bb338d5e41a443"
server_key_iv: "d619f1e04ea40ab7b27c46e6b3ecda0a"
server_key_version: "encryptionkey@0"
I20260928 20:19:29.735133 4231 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:19:29.735986 4246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:29.736244 4231 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:19:29.736352 4231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/master-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/master-0/wal
uuid: "3b2eb202db314d8e8d097278b55d724d"
format_stamp: "Formatted at 2026-09-28 20:19:29 on dist-test-slave-b83v"
server_key: "733bc8f92d5e2a2e99bb338d5e41a443"
server_key_iv: "d619f1e04ea40ab7b27c46e6b3ecda0a"
server_key_version: "encryptionkey@0"
I20260928 20:19:29.736434 4231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:19:29.763509 4231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:19:29.766019 4231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:19:29.766227 4231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:19:29.771050 4231 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.62:40329
I20260928 20:19:29.771096 4308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.62:40329 every 8 connection(s)
I20260928 20:19:29.771446 4231 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
I20260928 20:19:29.772135 4309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:29.774434 4309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b2eb202db314d8e8d097278b55d724d: Bootstrap starting.
I20260928 20:19:29.774617 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 4231
I20260928 20:19:29.774739 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/master-0/wal/instance
I20260928 20:19:29.774896 3144 external_mini_cluster.cc:1428] Setting key 5911e2d307740004b39119a7746b8e69
I20260928 20:19:29.775187 4309 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b2eb202db314d8e8d097278b55d724d: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:29.775568 4309 log.cc:824] T 00000000000000000000000000000000 P 3b2eb202db314d8e8d097278b55d724d: Log is configured to *not* fsync() on all Append() calls
I20260928 20:19:29.776368 4309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b2eb202db314d8e8d097278b55d724d: No bootstrap required, opened a new log
I20260928 20:19:29.778687 4309 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b2eb202db314d8e8d097278b55d724d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b2eb202db314d8e8d097278b55d724d" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 40329 } }
I20260928 20:19:29.778883 4309 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b2eb202db314d8e8d097278b55d724d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:29.778921 4309 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b2eb202db314d8e8d097278b55d724d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b2eb202db314d8e8d097278b55d724d, State: Initialized, Role: FOLLOWER
I20260928 20:19:29.779044 4309 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b2eb202db314d8e8d097278b55d724d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b2eb202db314d8e8d097278b55d724d" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 40329 } }
I20260928 20:19:29.779117 4309 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b2eb202db314d8e8d097278b55d724d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:19:29.779145 4309 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b2eb202db314d8e8d097278b55d724d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:19:29.779184 4309 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b2eb202db314d8e8d097278b55d724d [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:29.779911 4309 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b2eb202db314d8e8d097278b55d724d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b2eb202db314d8e8d097278b55d724d" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 40329 } }
I20260928 20:19:29.780000 4309 leader_election.cc:302] T 00000000000000000000000000000000 P 3b2eb202db314d8e8d097278b55d724d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b2eb202db314d8e8d097278b55d724d; no voters:
I20260928 20:19:29.780247 4309 leader_election.cc:288] T 00000000000000000000000000000000 P 3b2eb202db314d8e8d097278b55d724d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:19:29.780274 4314 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b2eb202db314d8e8d097278b55d724d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:19:29.780400 4314 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b2eb202db314d8e8d097278b55d724d [term 1 LEADER]: Becoming Leader. State: Replica: 3b2eb202db314d8e8d097278b55d724d, State: Running, Role: LEADER
I20260928 20:19:29.780529 4309 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b2eb202db314d8e8d097278b55d724d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:19:29.780555 4314 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b2eb202db314d8e8d097278b55d724d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b2eb202db314d8e8d097278b55d724d" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 40329 } }
I20260928 20:19:29.780925 4316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b2eb202db314d8e8d097278b55d724d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b2eb202db314d8e8d097278b55d724d. Latest consensus state: current_term: 1 leader_uuid: "3b2eb202db314d8e8d097278b55d724d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b2eb202db314d8e8d097278b55d724d" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 40329 } } }
I20260928 20:19:29.780933 4315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b2eb202db314d8e8d097278b55d724d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b2eb202db314d8e8d097278b55d724d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b2eb202db314d8e8d097278b55d724d" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 40329 } } }
I20260928 20:19:29.781114 4315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b2eb202db314d8e8d097278b55d724d [sys.catalog]: This master's current role is: LEADER
I20260928 20:19:29.781325 4316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b2eb202db314d8e8d097278b55d724d [sys.catalog]: This master's current role is: LEADER
I20260928 20:19:29.781581 4323 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:19:29.782079 4323 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 20:19:29 dist-test-slave-b83v krb5kdc[4218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:19:29.783865 4323 catalog_manager.cc:1347] Generated new cluster ID: cc9c454178554147a190bcc825fd5cbb
I20260928 20:19:29.783917 4323 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:19:29.787495 4312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:29.775550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51770 (local address 127.3.18.62:40329)
0928 20:19:29.775806 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:29.775817 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:29.775855 (+ 38us) server_negotiation.cc:407] Connection header received
0928 20:19:29.776356 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:29.776363 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:29.776413 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:29.776584 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:29.777374 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:29.778457 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:29.779497 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:29.780019 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:29.780184 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:29.784268 (+ 4084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:29.784273 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:29.784273 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:29.784285 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:29.785952 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:29.786339 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:29.786343 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:29.786343 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:29.786438 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:29.786660 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:29.786661 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:29.786661 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:29.786908 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:29.787084 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:29.787178 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 20:19:29.787326 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":92,"threads_started":1}
I20260928 20:19:29.812402 4323 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:19:29.812945 4323 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:19:29.817044 4323 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b2eb202db314d8e8d097278b55d724d: Generated new TSK 0
I20260928 20:19:29.817241 4323 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.18.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab.
Entry for principal kudu/127.3.18.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab.
I20260928 20:19:29.856058 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.1:0
--local_ip_for_outbound_sockets=127.3.18.1
--webserver_interface=127.3.18.1
--webserver_port=0
--tserver_master_addrs=127.3.18.62:40329
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab
--principal=kudu/127.3.18.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:45961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:19:29.945272 4335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:19:29.945535 4335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:19:29.945566 4335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:19:29.946870 4335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:19:29.946935 4335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:19:29.946969 4335 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 20:19:29.946990 4335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.1
I20260928 20:19:29.948581 4335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:45961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab
--principal=kudu/127.3.18.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.18.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.18.62:40329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.1
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:19:29.948828 4335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:19:29.949047 4335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:19:29.955629 4340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:29.955623 4343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:29.955677 4341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:29.955842 4335 server_base.cc:1034] running on GCE node
I20260928 20:19:29.956233 4335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:19:29.956514 4335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:19:29.957700 4335 hybrid_clock.cc:648] HybridClock initialized: now 1790626769957667 us; error 59 us; skew 500 ppm
Sep 28 20:19:29 dist-test-slave-b83v krb5kdc[4218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626769, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:19:29.960567 4335 init.cc:377] Logged in from keytab as kudu/127.3.18.1@KRBTEST.COM (short username kudu)
I20260928 20:19:29.961053 4335 webserver.cc:460] Webserver started at http://127.3.18.1:38209/ using document root <none> and password file <none>
I20260928 20:19:29.961270 4335 fs_manager.cc:359] Metadata directory not provided
I20260928 20:19:29.961316 4335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:19:29.961433 4335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:19:29.962555 4335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/ts-0/data/instance:
uuid: "b3734cff4fd64fce9d89707d7d6e9e94"
format_stamp: "Formatted at 2026-09-28 20:19:29 on dist-test-slave-b83v"
server_key: "9604a50e9939ab3f6bf81952b3277aea"
server_key_iv: "65555029cac7453423bd84bc89f18989"
server_key_version: "encryptionkey@0"
I20260928 20:19:29.962908 4335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/ts-0/wal/instance:
uuid: "b3734cff4fd64fce9d89707d7d6e9e94"
format_stamp: "Formatted at 2026-09-28 20:19:29 on dist-test-slave-b83v"
server_key: "9604a50e9939ab3f6bf81952b3277aea"
server_key_iv: "65555029cac7453423bd84bc89f18989"
server_key_version: "encryptionkey@0"
I20260928 20:19:29.964500 4335 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 20:19:29.965533 4350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:29.965751 4335 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:19:29.965853 4335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/ts-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/ts-0/wal
uuid: "b3734cff4fd64fce9d89707d7d6e9e94"
format_stamp: "Formatted at 2026-09-28 20:19:29 on dist-test-slave-b83v"
server_key: "9604a50e9939ab3f6bf81952b3277aea"
server_key_iv: "65555029cac7453423bd84bc89f18989"
server_key_version: "encryptionkey@0"
I20260928 20:19:29.965924 4335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:19:29.979460 4335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:19:29.982229 4335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:19:29.982358 4335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:19:29.982908 4335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:19:29.982949 4335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:29.982972 4335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:19:29.982985 4335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:29.991405 4335 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.1:40669
I20260928 20:19:29.991529 4465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.1:40669 every 8 connection(s)
Sep 28 20:19:29 dist-test-slave-b83v krb5kdc[4218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626769, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.1@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:19:29.991966 4335 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
I20260928 20:19:29.995247 4312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:29.983680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:33483 (local address 127.3.18.62:40329)
0928 20:19:29.983728 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:29.983730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:29.984237 (+ 507us) server_negotiation.cc:407] Connection header received
0928 20:19:29.985077 (+ 840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:29.985079 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:29.985097 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:29.985197 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:29.986170 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:29.987014 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:29.988817 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:29.989197 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:29.989296 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:29.992317 (+ 3021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:29.992319 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:29.992319 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:29.992328 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:29.993993 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:29.994425 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:29.994426 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:29.994426 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:29.994479 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:29.994689 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:29.994690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:29.994690 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:29.994791 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:29.994836 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:29.995094 (+ 258us) server_negotiation.cc:299] Negotiation successful
0928 20:19:29.995164 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 20:19:29.995384 4359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:29.983654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40329 (local address 127.3.18.1:33483)
0928 20:19:29.983876 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:29.984025 (+ 149us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:29.984896 (+ 871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:29.985301 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:29.985306 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:29.985693 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:29.986033 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:29.986039 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:29.987150 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:29.987152 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:29.988700 (+ 1548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:29.988704 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:29.989316 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:29.989317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:29.989488 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:29.990123 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:19:29.990133 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:29.992185 (+ 2052us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:29.994088 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:29.994094 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:29.994096 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:29.994350 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:29.994526 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:29.994527 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:29.994527 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:29.994622 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:29.994829 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:29.994832 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:29.994947 (+ 115us) client_negotiation.cc:769] Sending connection context
0928 20:19:29.995020 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 20:19:29.995144 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":138,"threads_started":1}
I20260928 20:19:29.995918 4466 heartbeater.cc:347] Connected to a master server at 127.3.18.62:40329
I20260928 20:19:29.996006 4466 heartbeater.cc:464] Registering TS with master...
I20260928 20:19:29.996212 4466 heartbeater.cc:511] Master 127.3.18.62:40329 requested a full tablet report, sending...
I20260928 20:19:29.996678 4262 ts_manager.cc:194] Registered new tserver with Master: b3734cff4fd64fce9d89707d7d6e9e94 (127.3.18.1:40669)
I20260928 20:19:29.997359 4262 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.18.1@KRBTEST.COM'} at 127.3.18.1:33483
I20260928 20:19:30.001668 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 4335
I20260928 20:19:30.001803 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/ts-0/wal/instance
I20260928 20:19:30.001995 3144 external_mini_cluster.cc:1428] Setting key bc2e8f24b313811541d23378990d50c0
I20260928 20:19:30.002712 4312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:29.997107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:53333 (local address 127.3.18.62:40329)
0928 20:19:29.997150 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:29.997152 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:29.997256 (+ 104us) server_negotiation.cc:407] Connection header received
0928 20:19:29.997328 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:29.997329 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:29.997339 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:29.997395 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:29.997717 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:29.998199 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:29.998893 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:29.999095 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:29.999177 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:30.000161 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:30.000162 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:30.000163 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:30.000169 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:30.001375 (+ 1206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:30.001729 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:30.001730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:30.001730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:30.001785 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:30.002042 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:30.002043 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:30.002043 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:30.002219 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:30.002293 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:30.002528 (+ 235us) server_negotiation.cc:299] Negotiation successful
0928 20:19:30.002620 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 20:19:30.002737 4472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:29.997047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40329 (local address 127.3.18.1:53333)
0928 20:19:29.997188 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:29.997197 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:29.997271 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:29.997400 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:29.997401 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:29.997590 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:29.997646 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:29.997648 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:29.998290 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:29.998291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:29.998803 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:29.998804 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:29.999177 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:29.999178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:29.999251 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:29.999637 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:19:29.999642 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:30.000003 (+ 361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:30.001460 (+ 1457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:30.001461 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:30.001462 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:30.001643 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:30.001847 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:30.001848 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:30.001848 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:30.001934 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:30.002332 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:30.002333 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:30.002374 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 20:19:30.002495 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 20:19:30.002618 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":78,"threads_started":1}
WARNING: no policy specified for kudu/127.3.18.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab.
Entry for principal kudu/127.3.18.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab.
I20260928 20:19:30.031478 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.2:0
--local_ip_for_outbound_sockets=127.3.18.2
--webserver_interface=127.3.18.2
--webserver_port=0
--tserver_master_addrs=127.3.18.62:40329
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab
--principal=kudu/127.3.18.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:45961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:19:30.119437 4475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:19:30.119685 4475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:19:30.119714 4475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:19:30.121071 4475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:19:30.121135 4475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:19:30.121160 4475 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 20:19:30.121178 4475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.2
I20260928 20:19:30.122810 4475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:45961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab
--principal=kudu/127.3.18.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.18.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.18.62:40329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.2
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:19:30.123091 4475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:19:30.123350 4475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:19:30.129005 4483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:30.129097 4481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:30.129005 4480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:30.129065 4475 server_base.cc:1034] running on GCE node
I20260928 20:19:30.129570 4475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:19:30.129856 4475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:19:30.131091 4475 hybrid_clock.cc:648] HybridClock initialized: now 1790626770131042 us; error 66 us; skew 500 ppm
Sep 28 20:19:30 dist-test-slave-b83v krb5kdc[4218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626770, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:19:30.134732 4475 init.cc:377] Logged in from keytab as kudu/127.3.18.2@KRBTEST.COM (short username kudu)
I20260928 20:19:30.135317 4475 webserver.cc:460] Webserver started at http://127.3.18.2:42623/ using document root <none> and password file <none>
I20260928 20:19:30.135569 4475 fs_manager.cc:359] Metadata directory not provided
I20260928 20:19:30.135644 4475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:19:30.135869 4475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:19:30.137173 4475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/ts-1/data/instance:
uuid: "13dd5ff0fc59493a84afa5ed4f71c1b0"
format_stamp: "Formatted at 2026-09-28 20:19:30 on dist-test-slave-b83v"
server_key: "e3998c44146164e33cda9b09d2f8e85a"
server_key_iv: "4675a98a6761a9a36e558fe32c6dc554"
server_key_version: "encryptionkey@0"
I20260928 20:19:30.137616 4475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/ts-1/wal/instance:
uuid: "13dd5ff0fc59493a84afa5ed4f71c1b0"
format_stamp: "Formatted at 2026-09-28 20:19:30 on dist-test-slave-b83v"
server_key: "e3998c44146164e33cda9b09d2f8e85a"
server_key_iv: "4675a98a6761a9a36e558fe32c6dc554"
server_key_version: "encryptionkey@0"
I20260928 20:19:30.139272 4475 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 20:19:30.140621 4490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:30.140897 4475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:19:30.141088 4475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/ts-1/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/ts-1/wal
uuid: "13dd5ff0fc59493a84afa5ed4f71c1b0"
format_stamp: "Formatted at 2026-09-28 20:19:30 on dist-test-slave-b83v"
server_key: "e3998c44146164e33cda9b09d2f8e85a"
server_key_iv: "4675a98a6761a9a36e558fe32c6dc554"
server_key_version: "encryptionkey@0"
I20260928 20:19:30.141240 4475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:19:30.158231 4475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:19:30.161095 4475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:19:30.161392 4475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:19:30.162078 4475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:19:30.162144 4475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:30.162181 4475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:19:30.162204 4475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 20:19:30 dist-test-slave-b83v krb5kdc[4218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626770, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.2@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:19:30.170961 4475 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.2:33777
I20260928 20:19:30.171471 4475 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
I20260928 20:19:30.172148 4605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.2:33777 every 8 connection(s)
I20260928 20:19:30.173761 4312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:30.162654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:52683 (local address 127.3.18.62:40329)
0928 20:19:30.162717 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:30.162719 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:30.162891 (+ 172us) server_negotiation.cc:407] Connection header received
0928 20:19:30.163418 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:30.163420 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:30.163437 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:30.163529 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:30.164546 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.165086 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.166301 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.166566 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.166664 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:30.169748 (+ 3084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:30.169750 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:30.169750 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:30.169759 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:30.171880 (+ 2121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:30.172540 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:30.172542 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:30.172542 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:30.172632 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:30.172968 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:30.172970 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:30.172971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:30.173108 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:30.173241 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:30.173587 (+ 346us) server_negotiation.cc:299] Negotiation successful
0928 20:19:30.173692 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 20:19:30.173890 4499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:30.162546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40329 (local address 127.3.18.2:52683)
0928 20:19:30.162736 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:30.162781 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:30.163305 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:30.163655 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:30.163661 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:30.164051 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:30.164428 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:30.164442 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.165199 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.165201 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:30.166136 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:30.166139 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.166677 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.166678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:30.166865 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:30.167534 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:19:30.167546 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:30.169565 (+ 2019us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:30.171994 (+ 2429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:30.171999 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:30.172001 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:30.172401 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:30.172724 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:30.172725 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:30.172725 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:30.172867 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:30.173257 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:30.173263 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:30.173407 (+ 144us) client_negotiation.cc:769] Sending connection context
0928 20:19:30.173505 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 20:19:30.173664 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":106,"threads_started":1}
I20260928 20:19:30.174460 4606 heartbeater.cc:347] Connected to a master server at 127.3.18.62:40329
I20260928 20:19:30.174551 4606 heartbeater.cc:464] Registering TS with master...
I20260928 20:19:30.174775 4606 heartbeater.cc:511] Master 127.3.18.62:40329 requested a full tablet report, sending...
I20260928 20:19:30.175202 4262 ts_manager.cc:194] Registered new tserver with Master: 13dd5ff0fc59493a84afa5ed4f71c1b0 (127.3.18.2:33777)
I20260928 20:19:30.175678 4262 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.18.2@KRBTEST.COM'} at 127.3.18.2:52683
I20260928 20:19:30.177253 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 4475
I20260928 20:19:30.177342 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/ts-1/wal/instance
I20260928 20:19:30.177471 3144 external_mini_cluster.cc:1428] Setting key c9b3a66e3e4b4ec916f0b123f8d2c270
I20260928 20:19:30.182953 4312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:30.175629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:42645 (local address 127.3.18.62:40329)
0928 20:19:30.175675 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:30.175677 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:30.175846 (+ 169us) server_negotiation.cc:407] Connection header received
0928 20:19:30.175912 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:30.175913 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:30.175923 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:30.175992 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:30.176420 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.176897 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.177880 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.178123 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.178187 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:30.179736 (+ 1549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:30.179738 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:30.179739 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:30.179749 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:30.181549 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:30.181939 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:30.181940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:30.181941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:30.182008 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:30.182327 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:30.182328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:30.182329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:30.182509 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:30.182594 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:30.182760 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 20:19:30.182850 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 20:19:30.182965 4612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:30.175568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40329 (local address 127.3.18.2:42645)
0928 20:19:30.175752 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:30.175797 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:30.175860 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:30.175994 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:30.175994 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:30.176231 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:30.176306 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:30.176308 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.177030 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.177031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:30.177800 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:30.177803 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.178251 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.178252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:30.178336 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:30.178972 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:19:30.178977 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:30.179460 (+ 483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:30.181643 (+ 2183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:30.181645 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:30.181645 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:30.181835 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:30.182163 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:30.182165 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:30.182165 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:30.182250 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:30.182589 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:30.182590 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:30.182643 (+ 53us) client_negotiation.cc:769] Sending connection context
0928 20:19:30.182729 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 20:19:30.182850 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":115,"threads_started":1}
WARNING: no policy specified for kudu/127.3.18.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab.
Entry for principal kudu/127.3.18.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab.
I20260928 20:19:30.208443 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.3:0
--local_ip_for_outbound_sockets=127.3.18.3
--webserver_interface=127.3.18.3
--webserver_port=0
--tserver_master_addrs=127.3.18.62:40329
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab
--principal=kudu/127.3.18.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:45961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:19:30.300431 4615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:19:30.300684 4615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:19:30.300724 4615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:19:30.302016 4615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:19:30.302089 4615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:19:30.302125 4615 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 20:19:30.302170 4615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.3
I20260928 20:19:30.304062 4615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:45961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab
--principal=kudu/127.3.18.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.18.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.18.62:40329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.3
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:19:30.304363 4615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:19:30.304615 4615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:19:30.309820 4620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:30.309839 4621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:30.309820 4623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:30.310166 4615 server_base.cc:1034] running on GCE node
I20260928 20:19:30.310391 4615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:19:30.310653 4615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:19:30.311846 4615 hybrid_clock.cc:648] HybridClock initialized: now 1790626770311822 us; error 35 us; skew 500 ppm
Sep 28 20:19:30 dist-test-slave-b83v krb5kdc[4218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626770, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:19:30.319573 4615 init.cc:377] Logged in from keytab as kudu/127.3.18.3@KRBTEST.COM (short username kudu)
I20260928 20:19:30.320164 4615 webserver.cc:460] Webserver started at http://127.3.18.3:34889/ using document root <none> and password file <none>
I20260928 20:19:30.320387 4615 fs_manager.cc:359] Metadata directory not provided
I20260928 20:19:30.320432 4615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:19:30.320528 4615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:19:30.321611 4615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/ts-2/data/instance:
uuid: "8b1a6b51b8344a45a8f9918d5130860e"
format_stamp: "Formatted at 2026-09-28 20:19:30 on dist-test-slave-b83v"
server_key: "f5d80e172dec6e3eb3bc7591dc755dc3"
server_key_iv: "97808d0c376de3f85b7cf66289db6f8e"
server_key_version: "encryptionkey@0"
I20260928 20:19:30.321934 4615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/ts-2/wal/instance:
uuid: "8b1a6b51b8344a45a8f9918d5130860e"
format_stamp: "Formatted at 2026-09-28 20:19:30 on dist-test-slave-b83v"
server_key: "f5d80e172dec6e3eb3bc7591dc755dc3"
server_key_iv: "97808d0c376de3f85b7cf66289db6f8e"
server_key_version: "encryptionkey@0"
I20260928 20:19:30.323491 4615 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 20:19:30.324515 4630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:30.324777 4615 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:19:30.324898 4615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/ts-2/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/ts-2/wal
uuid: "8b1a6b51b8344a45a8f9918d5130860e"
format_stamp: "Formatted at 2026-09-28 20:19:30 on dist-test-slave-b83v"
server_key: "f5d80e172dec6e3eb3bc7591dc755dc3"
server_key_iv: "97808d0c376de3f85b7cf66289db6f8e"
server_key_version: "encryptionkey@0"
I20260928 20:19:30.325001 4615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:19:30.344760 4615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:19:30.347347 4615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:19:30.347471 4615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:19:30.348114 4615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:19:30.348160 4615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:30.348182 4615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:19:30.348196 4615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:30.355463 4615 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.3:46789
I20260928 20:19:30.355695 4745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.3:46789 every 8 connection(s)
Sep 28 20:19:30 dist-test-slave-b83v krb5kdc[4218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626770, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.3@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:19:30.356021 4615 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
I20260928 20:19:30.359932 4312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:30.348603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:45469 (local address 127.3.18.62:40329)
0928 20:19:30.348680 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:30.348683 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:30.348916 (+ 233us) server_negotiation.cc:407] Connection header received
0928 20:19:30.349810 (+ 894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:30.349812 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:30.349830 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:30.349908 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:30.350904 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.351839 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.353037 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.353384 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.353478 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:30.356627 (+ 3149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:30.356630 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:30.356631 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:30.356639 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:30.358112 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:30.358632 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:30.358634 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:30.358634 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:30.358698 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:30.358999 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:30.359000 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:30.359000 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:30.359135 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:30.359203 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:30.359696 (+ 493us) server_negotiation.cc:299] Negotiation successful
0928 20:19:30.359855 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 20:19:30.360136 4639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:30.348586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40329 (local address 127.3.18.3:45469)
0928 20:19:30.348773 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:30.348818 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:30.349648 (+ 830us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:30.350028 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:30.350034 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:30.350445 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:30.350780 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:30.350788 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.351943 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.351944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:30.352941 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:30.352944 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.353489 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.353490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:30.353634 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:30.354332 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:19:30.354342 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:30.356458 (+ 2116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:30.358290 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:30.358298 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:30.358300 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:30.358543 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:30.358789 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:30.358790 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:30.358790 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:30.358910 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:30.359385 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:30.359391 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:30.359527 (+ 136us) client_negotiation.cc:769] Sending connection context
0928 20:19:30.359643 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 20:19:30.359876 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":120,"threads_started":1}
I20260928 20:19:30.360649 4746 heartbeater.cc:347] Connected to a master server at 127.3.18.62:40329
I20260928 20:19:30.360730 4746 heartbeater.cc:464] Registering TS with master...
I20260928 20:19:30.360939 4746 heartbeater.cc:511] Master 127.3.18.62:40329 requested a full tablet report, sending...
I20260928 20:19:30.361368 4263 ts_manager.cc:194] Registered new tserver with Master: 8b1a6b51b8344a45a8f9918d5130860e (127.3.18.3:46789)
I20260928 20:19:30.361819 4263 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.18.3@KRBTEST.COM'} at 127.3.18.3:45469
I20260928 20:19:30.364494 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 4615
I20260928 20:19:30.364588 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/ts-2/wal/instance
I20260928 20:19:30.364833 3144 external_mini_cluster.cc:1428] Setting key dff2243d07c6441499965fbbf65f77e9
I20260928 20:19:30.365803 3144 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 20:19:30.368731 4312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:30.362176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:48103 (local address 127.3.18.62:40329)
0928 20:19:30.362229 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:30.362231 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:30.362258 (+ 27us) server_negotiation.cc:407] Connection header received
0928 20:19:30.362381 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:30.362382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:30.362392 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:30.362442 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:30.362792 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.363429 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.364317 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.364540 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.364619 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:30.365936 (+ 1317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:30.365937 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:30.365938 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:30.365947 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:30.367382 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:30.367802 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:30.367804 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:30.367804 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:30.367874 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:30.368111 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:30.368112 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:30.368112 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:30.368261 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:30.368339 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:30.368514 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 20:19:30.368624 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 20:19:30.368748 4752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:30.361973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40329 (local address 127.3.18.3:48103)
0928 20:19:30.362197 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:30.362209 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:30.362312 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:30.362445 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:30.362446 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:30.362657 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:30.362715 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:30.362717 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.363551 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.363552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:30.364213 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:30.364216 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.364686 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.364686 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:30.364763 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:30.365340 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:19:30.365345 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:30.365844 (+ 499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:30.367481 (+ 1637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:30.367482 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:30.367483 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:30.367678 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:30.367948 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:30.367950 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:30.367951 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:30.368017 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:30.368340 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:30.368341 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:30.368391 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 20:19:30.368483 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 20:19:30.368633 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":105,"threads_started":1}
Sep 28 20:19:30 dist-test-slave-b83v krb5kdc[4218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626770, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 20:19:30 dist-test-slave-b83v krb5kdc[4218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626770, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:19:30.391049 4312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:30.383136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51784 (local address 127.3.18.62:40329)
0928 20:19:30.383237 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:30.383239 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:30.383251 (+ 12us) server_negotiation.cc:407] Connection header received
0928 20:19:30.383314 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:30.383315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:30.383325 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:30.383384 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:30.384164 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.384682 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.385510 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.385719 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.385785 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:30.388429 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:30.388432 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:30.388432 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:30.388442 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:30.389964 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:30.390314 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:30.390315 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:30.390315 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:30.390371 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:30.390545 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:30.390546 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:30.390547 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:30.390707 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:30.390791 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:30.390911 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 20:19:30.390976 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 20:19:30.393139 4263 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51784:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 20:19:30.394850 4263 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 20:19:30.401499 4766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:30.397339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52246 (local address 127.3.18.2:33777)
0928 20:19:30.397589 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:30.397592 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:30.397605 (+ 13us) server_negotiation.cc:407] Connection header received
0928 20:19:30.397769 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:30.397772 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:30.397790 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:30.397859 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:19:30.398118 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.399248 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.400530 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.401022 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.401118 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:30.401180 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:30.401353 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 20:19:30.401398 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":83,"threads_started":1}
I20260928 20:19:30.401496 4763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:30.397227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.2:33777 (local address 127.0.0.1:52246)
0928 20:19:30.397513 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:30.397558 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:30.397629 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:30.397934 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:30.397937 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:30.397939 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:19:30.398032 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:30.398035 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.399345 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.399346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:30.400453 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:30.400455 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.401125 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.401126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:30.401228 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:30.401243 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 20:19:30.401334 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 20:19:30.401399 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":56,"thread_start_us":96,"threads_started":1}
I20260928 20:19:30.401546 4765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:30.397327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37608 (local address 127.3.18.3:46789)
0928 20:19:30.397511 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:30.397516 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:30.397726 (+ 210us) server_negotiation.cc:407] Connection header received
0928 20:19:30.397867 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:30.397870 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:30.397887 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:30.398028 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:19:30.398172 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.398866 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.400288 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.401037 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.401111 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:30.401180 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:30.401388 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 20:19:30.401434 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":78,"threads_started":1}
I20260928 20:19:30.401786 4767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:30.397417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.1:40669 (local address 127.0.0.1:47478)
0928 20:19:30.397684 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:30.397695 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:30.397749 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:30.397934 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:30.397937 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:30.397941 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:19:30.398144 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:30.398145 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.399382 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.399383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:30.400421 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:30.400424 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.401357 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.401358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:30.401409 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:30.401412 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 20:19:30.401454 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 20:19:30.401494 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":142,"thread_start_us":68,"threads_started":1}
I20260928 20:19:30.401813 4768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:30.397511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47478 (local address 127.3.18.1:40669)
0928 20:19:30.397667 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:30.397672 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:30.397733 (+ 61us) server_negotiation.cc:407] Connection header received
0928 20:19:30.397819 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:30.397822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:30.397838 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:30.397919 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:19:30.398225 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.399298 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.400518 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.401217 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.401277 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:30.401326 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:30.401536 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 20:19:30.401581 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":91,"threads_started":1}
I20260928 20:19:30.401790 4764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:30.397229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.3:46789 (local address 127.0.0.1:37608)
0928 20:19:30.397664 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:30.397673 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:30.397753 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:30.398020 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:30.398021 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:30.398022 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:19:30.398057 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:30.398058 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.398953 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.398954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:30.400199 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:30.400201 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.401220 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.401221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:30.401291 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:30.401295 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:19:30.401342 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 20:19:30.401706 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"mutex_wait_us":311,"thread_start_us":47,"threads_started":1}
I20260928 20:19:30.402771 4400 tablet_service.cc:1467] Processing CreateTablet for tablet b606ad67275942289baa32313068d483 (DEFAULT_TABLE table=test-table [id=d72854ec94a545aea1e37cdf1d48ae50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:19:30.402771 4680 tablet_service.cc:1467] Processing CreateTablet for tablet b606ad67275942289baa32313068d483 (DEFAULT_TABLE table=test-table [id=d72854ec94a545aea1e37cdf1d48ae50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:19:30.403165 4400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b606ad67275942289baa32313068d483. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:30.403165 4680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b606ad67275942289baa32313068d483. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:30.404384 4540 tablet_service.cc:1467] Processing CreateTablet for tablet b606ad67275942289baa32313068d483 (DEFAULT_TABLE table=test-table [id=d72854ec94a545aea1e37cdf1d48ae50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:19:30.404716 4540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b606ad67275942289baa32313068d483. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:30.406585 4769 tablet_bootstrap.cc:492] T b606ad67275942289baa32313068d483 P 8b1a6b51b8344a45a8f9918d5130860e: Bootstrap starting.
I20260928 20:19:30.407060 4770 tablet_bootstrap.cc:492] T b606ad67275942289baa32313068d483 P b3734cff4fd64fce9d89707d7d6e9e94: Bootstrap starting.
I20260928 20:19:30.407650 4770 tablet_bootstrap.cc:654] T b606ad67275942289baa32313068d483 P b3734cff4fd64fce9d89707d7d6e9e94: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:30.407655 4769 tablet_bootstrap.cc:654] T b606ad67275942289baa32313068d483 P 8b1a6b51b8344a45a8f9918d5130860e: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:30.407991 4771 tablet_bootstrap.cc:492] T b606ad67275942289baa32313068d483 P 13dd5ff0fc59493a84afa5ed4f71c1b0: Bootstrap starting.
I20260928 20:19:30.408059 4770 log.cc:824] T b606ad67275942289baa32313068d483 P b3734cff4fd64fce9d89707d7d6e9e94: Log is configured to *not* fsync() on all Append() calls
I20260928 20:19:30.408084 4769 log.cc:824] T b606ad67275942289baa32313068d483 P 8b1a6b51b8344a45a8f9918d5130860e: Log is configured to *not* fsync() on all Append() calls
I20260928 20:19:30.408748 4771 tablet_bootstrap.cc:654] T b606ad67275942289baa32313068d483 P 13dd5ff0fc59493a84afa5ed4f71c1b0: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:30.409003 4769 tablet_bootstrap.cc:492] T b606ad67275942289baa32313068d483 P 8b1a6b51b8344a45a8f9918d5130860e: No bootstrap required, opened a new log
I20260928 20:19:30.409088 4769 ts_tablet_manager.cc:1397] T b606ad67275942289baa32313068d483 P 8b1a6b51b8344a45a8f9918d5130860e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 20:19:30.409081 4771 log.cc:824] T b606ad67275942289baa32313068d483 P 13dd5ff0fc59493a84afa5ed4f71c1b0: Log is configured to *not* fsync() on all Append() calls
I20260928 20:19:30.409826 4771 tablet_bootstrap.cc:492] T b606ad67275942289baa32313068d483 P 13dd5ff0fc59493a84afa5ed4f71c1b0: No bootstrap required, opened a new log
I20260928 20:19:30.409876 4770 tablet_bootstrap.cc:492] T b606ad67275942289baa32313068d483 P b3734cff4fd64fce9d89707d7d6e9e94: No bootstrap required, opened a new log
I20260928 20:19:30.409909 4771 ts_tablet_manager.cc:1397] T b606ad67275942289baa32313068d483 P 13dd5ff0fc59493a84afa5ed4f71c1b0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 20:19:30.409930 4770 ts_tablet_manager.cc:1397] T b606ad67275942289baa32313068d483 P b3734cff4fd64fce9d89707d7d6e9e94: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260928 20:19:30.410732 4769 raft_consensus.cc:348] T b606ad67275942289baa32313068d483 P 8b1a6b51b8344a45a8f9918d5130860e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b1a6b51b8344a45a8f9918d5130860e" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 46789 } } peers { permanent_uuid: "b3734cff4fd64fce9d89707d7d6e9e94" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40669 } } peers { permanent_uuid: "13dd5ff0fc59493a84afa5ed4f71c1b0" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 33777 } }
I20260928 20:19:30.410861 4769 raft_consensus.cc:374] T b606ad67275942289baa32313068d483 P 8b1a6b51b8344a45a8f9918d5130860e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:30.410882 4769 raft_consensus.cc:729] T b606ad67275942289baa32313068d483 P 8b1a6b51b8344a45a8f9918d5130860e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b1a6b51b8344a45a8f9918d5130860e, State: Initialized, Role: FOLLOWER
I20260928 20:19:30.410976 4769 consensus_queue.cc:260] T b606ad67275942289baa32313068d483 P 8b1a6b51b8344a45a8f9918d5130860e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b1a6b51b8344a45a8f9918d5130860e" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 46789 } } peers { permanent_uuid: "b3734cff4fd64fce9d89707d7d6e9e94" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40669 } } peers { permanent_uuid: "13dd5ff0fc59493a84afa5ed4f71c1b0" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 33777 } }
I20260928 20:19:30.411242 4769 ts_tablet_manager.cc:1428] T b606ad67275942289baa32313068d483 P 8b1a6b51b8344a45a8f9918d5130860e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 20:19:30.411245 4746 heartbeater.cc:503] Master 127.3.18.62:40329 was elected leader, sending a full tablet report...
I20260928 20:19:30.412021 4770 raft_consensus.cc:348] T b606ad67275942289baa32313068d483 P b3734cff4fd64fce9d89707d7d6e9e94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b1a6b51b8344a45a8f9918d5130860e" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 46789 } } peers { permanent_uuid: "b3734cff4fd64fce9d89707d7d6e9e94" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40669 } } peers { permanent_uuid: "13dd5ff0fc59493a84afa5ed4f71c1b0" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 33777 } }
I20260928 20:19:30.412009 4771 raft_consensus.cc:348] T b606ad67275942289baa32313068d483 P 13dd5ff0fc59493a84afa5ed4f71c1b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b1a6b51b8344a45a8f9918d5130860e" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 46789 } } peers { permanent_uuid: "b3734cff4fd64fce9d89707d7d6e9e94" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40669 } } peers { permanent_uuid: "13dd5ff0fc59493a84afa5ed4f71c1b0" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 33777 } }
I20260928 20:19:30.412173 4771 raft_consensus.cc:374] T b606ad67275942289baa32313068d483 P 13dd5ff0fc59493a84afa5ed4f71c1b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:30.412170 4770 raft_consensus.cc:374] T b606ad67275942289baa32313068d483 P b3734cff4fd64fce9d89707d7d6e9e94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:30.412209 4771 raft_consensus.cc:729] T b606ad67275942289baa32313068d483 P 13dd5ff0fc59493a84afa5ed4f71c1b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13dd5ff0fc59493a84afa5ed4f71c1b0, State: Initialized, Role: FOLLOWER
I20260928 20:19:30.412209 4770 raft_consensus.cc:729] T b606ad67275942289baa32313068d483 P b3734cff4fd64fce9d89707d7d6e9e94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3734cff4fd64fce9d89707d7d6e9e94, State: Initialized, Role: FOLLOWER
I20260928 20:19:30.412328 4770 consensus_queue.cc:260] T b606ad67275942289baa32313068d483 P b3734cff4fd64fce9d89707d7d6e9e94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b1a6b51b8344a45a8f9918d5130860e" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 46789 } } peers { permanent_uuid: "b3734cff4fd64fce9d89707d7d6e9e94" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40669 } } peers { permanent_uuid: "13dd5ff0fc59493a84afa5ed4f71c1b0" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 33777 } }
I20260928 20:19:30.412413 4771 consensus_queue.cc:260] T b606ad67275942289baa32313068d483 P 13dd5ff0fc59493a84afa5ed4f71c1b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b1a6b51b8344a45a8f9918d5130860e" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 46789 } } peers { permanent_uuid: "b3734cff4fd64fce9d89707d7d6e9e94" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40669 } } peers { permanent_uuid: "13dd5ff0fc59493a84afa5ed4f71c1b0" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 33777 } }
I20260928 20:19:30.412669 4466 heartbeater.cc:503] Master 127.3.18.62:40329 was elected leader, sending a full tablet report...
I20260928 20:19:30.412681 4771 ts_tablet_manager.cc:1428] T b606ad67275942289baa32313068d483 P 13dd5ff0fc59493a84afa5ed4f71c1b0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 20:19:30.412814 4606 heartbeater.cc:503] Master 127.3.18.62:40329 was elected leader, sending a full tablet report...
I20260928 20:19:30.412947 4770 ts_tablet_manager.cc:1428] T b606ad67275942289baa32313068d483 P b3734cff4fd64fce9d89707d7d6e9e94: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260928 20:19:30.413532 4776 raft_consensus.cc:482] T b606ad67275942289baa32313068d483 P b3734cff4fd64fce9d89707d7d6e9e94 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:19:30.413666 4776 raft_consensus.cc:504] T b606ad67275942289baa32313068d483 P b3734cff4fd64fce9d89707d7d6e9e94 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b1a6b51b8344a45a8f9918d5130860e" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 46789 } } peers { permanent_uuid: "b3734cff4fd64fce9d89707d7d6e9e94" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40669 } } peers { permanent_uuid: "13dd5ff0fc59493a84afa5ed4f71c1b0" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 33777 } }
I20260928 20:19:30.414037 4776 leader_election.cc:288] T b606ad67275942289baa32313068d483 P b3734cff4fd64fce9d89707d7d6e9e94 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b1a6b51b8344a45a8f9918d5130860e (127.3.18.3:46789), 13dd5ff0fc59493a84afa5ed4f71c1b0 (127.3.18.2:33777)
I20260928 20:19:30.417182 4766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:30.414373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:58381 (local address 127.3.18.2:33777)
0928 20:19:30.414416 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:30.414418 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:30.414427 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:19:30.414468 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:30.414469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:30.414481 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:30.414553 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:19:30.414689 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.415329 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.416351 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.416815 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.416874 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:30.416883 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:30.417062 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 20:19:30.417111 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 20:19:30.417213 4765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:30.414437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:48717 (local address 127.3.18.3:46789)
0928 20:19:30.414483 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:30.414485 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:30.414712 (+ 227us) server_negotiation.cc:407] Connection header received
0928 20:19:30.414784 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:30.414784 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:30.414792 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:30.414829 (+ 37us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:19:30.414976 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.415441 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.416505 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.416861 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.416927 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:30.416937 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:30.417119 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 20:19:30.417155 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 20:19:30.417147 4359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:30.414250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.2:33777 (local address 127.3.18.1:58381)
0928 20:19:30.414307 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:30.414319 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:30.414360 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:30.414555 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:30.414555 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:30.414557 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:19:30.414613 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:30.414614 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.415407 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.415408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:30.416268 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:30.416270 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.416888 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.416889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:30.416971 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:30.416975 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:19:30.417026 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 20:19:30.417059 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 20:19:30.417532 4560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b606ad67275942289baa32313068d483" candidate_uuid: "b3734cff4fd64fce9d89707d7d6e9e94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13dd5ff0fc59493a84afa5ed4f71c1b0" is_pre_election: true
I20260928 20:19:30.417673 4560 raft_consensus.cc:2454] T b606ad67275942289baa32313068d483 P 13dd5ff0fc59493a84afa5ed4f71c1b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3734cff4fd64fce9d89707d7d6e9e94 in term 0.
I20260928 20:19:30.417735 4778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:30.414256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.3:46789 (local address 127.3.18.1:48717)
0928 20:19:30.414632 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:30.414664 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:30.414725 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:30.414833 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:30.414834 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:30.414835 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:19:30.414915 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:30.414917 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.415516 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.415517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:30.416413 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:30.416415 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.416941 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.416943 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:30.417030 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:30.417035 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 20:19:30.417106 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 20:19:30.417147 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"mutex_wait_us":33,"thread_start_us":90,"threads_started":1}
I20260928 20:19:30.417927 4352 leader_election.cc:302] T b606ad67275942289baa32313068d483 P b3734cff4fd64fce9d89707d7d6e9e94 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13dd5ff0fc59493a84afa5ed4f71c1b0, b3734cff4fd64fce9d89707d7d6e9e94; no voters:
I20260928 20:19:30.418061 4776 raft_consensus.cc:2781] T b606ad67275942289baa32313068d483 P b3734cff4fd64fce9d89707d7d6e9e94 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 20:19:30.418071 4700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b606ad67275942289baa32313068d483" candidate_uuid: "b3734cff4fd64fce9d89707d7d6e9e94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b1a6b51b8344a45a8f9918d5130860e" is_pre_election: true
I20260928 20:19:30.418145 4776 raft_consensus.cc:482] T b606ad67275942289baa32313068d483 P b3734cff4fd64fce9d89707d7d6e9e94 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 20:19:30.418185 4776 raft_consensus.cc:3037] T b606ad67275942289baa32313068d483 P b3734cff4fd64fce9d89707d7d6e9e94 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:30.418227 4700 raft_consensus.cc:2454] T b606ad67275942289baa32313068d483 P 8b1a6b51b8344a45a8f9918d5130860e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3734cff4fd64fce9d89707d7d6e9e94 in term 0.
I20260928 20:19:30.419297 4776 raft_consensus.cc:504] T b606ad67275942289baa32313068d483 P b3734cff4fd64fce9d89707d7d6e9e94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b1a6b51b8344a45a8f9918d5130860e" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 46789 } } peers { permanent_uuid: "b3734cff4fd64fce9d89707d7d6e9e94" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40669 } } peers { permanent_uuid: "13dd5ff0fc59493a84afa5ed4f71c1b0" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 33777 } }
I20260928 20:19:30.419435 4776 leader_election.cc:288] T b606ad67275942289baa32313068d483 P b3734cff4fd64fce9d89707d7d6e9e94 [CANDIDATE]: Term 1 election: Requested vote from peers 8b1a6b51b8344a45a8f9918d5130860e (127.3.18.3:46789), 13dd5ff0fc59493a84afa5ed4f71c1b0 (127.3.18.2:33777)
I20260928 20:19:30.419593 4560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b606ad67275942289baa32313068d483" candidate_uuid: "b3734cff4fd64fce9d89707d7d6e9e94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13dd5ff0fc59493a84afa5ed4f71c1b0"
I20260928 20:19:30.419620 4700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b606ad67275942289baa32313068d483" candidate_uuid: "b3734cff4fd64fce9d89707d7d6e9e94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b1a6b51b8344a45a8f9918d5130860e"
I20260928 20:19:30.419657 4560 raft_consensus.cc:3037] T b606ad67275942289baa32313068d483 P 13dd5ff0fc59493a84afa5ed4f71c1b0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:30.419730 4700 raft_consensus.cc:3037] T b606ad67275942289baa32313068d483 P 8b1a6b51b8344a45a8f9918d5130860e [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:30.420413 4560 raft_consensus.cc:2454] T b606ad67275942289baa32313068d483 P 13dd5ff0fc59493a84afa5ed4f71c1b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3734cff4fd64fce9d89707d7d6e9e94 in term 1.
I20260928 20:19:30.420537 4700 raft_consensus.cc:2454] T b606ad67275942289baa32313068d483 P 8b1a6b51b8344a45a8f9918d5130860e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3734cff4fd64fce9d89707d7d6e9e94 in term 1.
I20260928 20:19:30.420567 4352 leader_election.cc:302] T b606ad67275942289baa32313068d483 P b3734cff4fd64fce9d89707d7d6e9e94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13dd5ff0fc59493a84afa5ed4f71c1b0, b3734cff4fd64fce9d89707d7d6e9e94; no voters:
I20260928 20:19:30.420696 4776 raft_consensus.cc:2781] T b606ad67275942289baa32313068d483 P b3734cff4fd64fce9d89707d7d6e9e94 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:19:30.420835 4776 raft_consensus.cc:686] T b606ad67275942289baa32313068d483 P b3734cff4fd64fce9d89707d7d6e9e94 [term 1 LEADER]: Becoming Leader. State: Replica: b3734cff4fd64fce9d89707d7d6e9e94, State: Running, Role: LEADER
I20260928 20:19:30.420933 4776 consensus_queue.cc:237] T b606ad67275942289baa32313068d483 P b3734cff4fd64fce9d89707d7d6e9e94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b1a6b51b8344a45a8f9918d5130860e" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 46789 } } peers { permanent_uuid: "b3734cff4fd64fce9d89707d7d6e9e94" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40669 } } peers { permanent_uuid: "13dd5ff0fc59493a84afa5ed4f71c1b0" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 33777 } }
I20260928 20:19:30.421922 4263 catalog_manager.cc:5495] T b606ad67275942289baa32313068d483 P b3734cff4fd64fce9d89707d7d6e9e94 reported cstate change: term changed from 0 to 1, leader changed from <none> to b3734cff4fd64fce9d89707d7d6e9e94 (127.3.18.1). New cstate: current_term: 1 leader_uuid: "b3734cff4fd64fce9d89707d7d6e9e94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b1a6b51b8344a45a8f9918d5130860e" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 46789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3734cff4fd64fce9d89707d7d6e9e94" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40669 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13dd5ff0fc59493a84afa5ed4f71c1b0" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 33777 } health_report { overall_health: UNKNOWN } } }
W20260928 20:19:30.422178 4607 tablet.cc:2369] T b606ad67275942289baa32313068d483 P 13dd5ff0fc59493a84afa5ed4f71c1b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 20:19:30.432368 4768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:30.428269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47482 (local address 127.3.18.1:40669)
0928 20:19:30.428339 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:30.428341 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:30.428354 (+ 13us) server_negotiation.cc:407] Connection header received
0928 20:19:30.428443 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:30.428445 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:30.428499 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:30.428588 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:19:30.428857 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.429466 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.430874 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.431190 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.431279 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:30.431505 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:19:30.431651 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:19:30.431781 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:30.432227 (+ 446us) server_negotiation.cc:299] Negotiation successful
0928 20:19:30.432259 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 20:19:30.434437 4560 raft_consensus.cc:1261] T b606ad67275942289baa32313068d483 P 13dd5ff0fc59493a84afa5ed4f71c1b0 [term 1 FOLLOWER]: Refusing update from remote peer b3734cff4fd64fce9d89707d7d6e9e94: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:19:30.435034 4779 consensus_queue.cc:1035] T b606ad67275942289baa32313068d483 P b3734cff4fd64fce9d89707d7d6e9e94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13dd5ff0fc59493a84afa5ed4f71c1b0" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 33777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:19:30.437441 4700 raft_consensus.cc:1261] T b606ad67275942289baa32313068d483 P 8b1a6b51b8344a45a8f9918d5130860e [term 1 FOLLOWER]: Refusing update from remote peer b3734cff4fd64fce9d89707d7d6e9e94: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:19:30.437902 4779 consensus_queue.cc:1035] T b606ad67275942289baa32313068d483 P b3734cff4fd64fce9d89707d7d6e9e94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b1a6b51b8344a45a8f9918d5130860e" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 46789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:19:30.438419 4782 mvcc.cc:205] Tried to move back new op lower bound from 7334407251696304128 to 7334407251644784640. Current Snapshot: MvccSnapshot[applied={T|T < 7334407251696304128}]
I20260928 20:19:30.440618 4783 mvcc.cc:205] Tried to move back new op lower bound from 7334407251696304128 to 7334407251644784640. Current Snapshot: MvccSnapshot[applied={T|T < 7334407251696304128}]
I20260928 20:19:30.442203 4787 mvcc.cc:205] Tried to move back new op lower bound from 7334407251696304128 to 7334407251644784640. Current Snapshot: MvccSnapshot[applied={T|T < 7334407251696304128}]
I20260928 20:19:30.443749 4261 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:19:30.443923 4261 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:19:30.445073 4261 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3b2eb202db314d8e8d097278b55d724d: Sending DeleteTablet for 3 replicas of tablet b606ad67275942289baa32313068d483
I20260928 20:19:30.445487 4540 tablet_service.cc:1514] Processing DeleteTablet for tablet b606ad67275942289baa32313068d483 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:19:30 UTC) from {username='kudu', principal='kudu/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:52246
I20260928 20:19:30.445485 4400 tablet_service.cc:1514] Processing DeleteTablet for tablet b606ad67275942289baa32313068d483 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:19:30 UTC) from {username='kudu', principal='kudu/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:47478
I20260928 20:19:30.445885 4680 tablet_service.cc:1514] Processing DeleteTablet for tablet b606ad67275942289baa32313068d483 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:19:30 UTC) from {username='kudu', principal='kudu/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:37608
I20260928 20:19:30.445989 4792 tablet_replica.cc:330] T b606ad67275942289baa32313068d483 P 13dd5ff0fc59493a84afa5ed4f71c1b0: stopping tablet replica
I20260928 20:19:30.446113 4792 raft_consensus.cc:2229] T b606ad67275942289baa32313068d483 P 13dd5ff0fc59493a84afa5ed4f71c1b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:19:30.446074 4261 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51784:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 20:19:30.446167 4794 tablet_replica.cc:330] T b606ad67275942289baa32313068d483 P 8b1a6b51b8344a45a8f9918d5130860e: stopping tablet replica
W20260928 20:19:30.446238 4261 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 20:19:30.446270 4794 raft_consensus.cc:2229] T b606ad67275942289baa32313068d483 P 8b1a6b51b8344a45a8f9918d5130860e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:19:30.446273 4792 raft_consensus.cc:2258] T b606ad67275942289baa32313068d483 P 13dd5ff0fc59493a84afa5ed4f71c1b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:19:30.446379 4794 raft_consensus.cc:2258] T b606ad67275942289baa32313068d483 P 8b1a6b51b8344a45a8f9918d5130860e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:19:30.446544 4793 tablet_replica.cc:330] T b606ad67275942289baa32313068d483 P b3734cff4fd64fce9d89707d7d6e9e94: stopping tablet replica
I20260928 20:19:30.446805 4792 ts_tablet_manager.cc:1905] T b606ad67275942289baa32313068d483 P 13dd5ff0fc59493a84afa5ed4f71c1b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:19:30.447085 4793 raft_consensus.cc:2229] T b606ad67275942289baa32313068d483 P b3734cff4fd64fce9d89707d7d6e9e94 [term 1 LEADER]: Raft consensus shutting down.
I20260928 20:19:30.447283 4793 raft_consensus.cc:2258] T b606ad67275942289baa32313068d483 P b3734cff4fd64fce9d89707d7d6e9e94 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:19:30.448278 4794 ts_tablet_manager.cc:1905] T b606ad67275942289baa32313068d483 P 8b1a6b51b8344a45a8f9918d5130860e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:19:30.448344 4540 tablet_service.cc:1467] Processing CreateTablet for tablet 38d17bed4b8a466b9be4c0eee1d8f5df (DEFAULT_TABLE table=test-table [id=fb5b21539dff412fbd81253697c3b220]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:19:30.448484 4540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38d17bed4b8a466b9be4c0eee1d8f5df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:30.449048 4680 tablet_service.cc:1467] Processing CreateTablet for tablet 38d17bed4b8a466b9be4c0eee1d8f5df (DEFAULT_TABLE table=test-table [id=fb5b21539dff412fbd81253697c3b220]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:19:30.449170 4680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38d17bed4b8a466b9be4c0eee1d8f5df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:30.450403 4769 tablet_bootstrap.cc:492] T 38d17bed4b8a466b9be4c0eee1d8f5df P 8b1a6b51b8344a45a8f9918d5130860e: Bootstrap starting.
I20260928 20:19:30.451084 4769 tablet_bootstrap.cc:654] T 38d17bed4b8a466b9be4c0eee1d8f5df P 8b1a6b51b8344a45a8f9918d5130860e: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:30.451221 4792 ts_tablet_manager.cc:1918] T b606ad67275942289baa32313068d483 P 13dd5ff0fc59493a84afa5ed4f71c1b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:19:30.451313 4792 log.cc:1196] T b606ad67275942289baa32313068d483 P 13dd5ff0fc59493a84afa5ed4f71c1b0: Deleting WAL directory at /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/ts-1/wal/wals/b606ad67275942289baa32313068d483
I20260928 20:19:30.451450 4794 ts_tablet_manager.cc:1918] T b606ad67275942289baa32313068d483 P 8b1a6b51b8344a45a8f9918d5130860e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:19:30.451550 4794 log.cc:1196] T b606ad67275942289baa32313068d483 P 8b1a6b51b8344a45a8f9918d5130860e: Deleting WAL directory at /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/ts-2/wal/wals/b606ad67275942289baa32313068d483
I20260928 20:19:30.451624 4792 ts_tablet_manager.cc:1939] T b606ad67275942289baa32313068d483 P 13dd5ff0fc59493a84afa5ed4f71c1b0: Deleting consensus metadata
I20260928 20:19:30.452013 4794 ts_tablet_manager.cc:1939] T b606ad67275942289baa32313068d483 P 8b1a6b51b8344a45a8f9918d5130860e: Deleting consensus metadata
I20260928 20:19:30.452576 4250 catalog_manager.cc:4841] TS 8b1a6b51b8344a45a8f9918d5130860e (127.3.18.3:46789): tablet b606ad67275942289baa32313068d483 (table test-table [id=d72854ec94a545aea1e37cdf1d48ae50]) successfully deleted
I20260928 20:19:30.453660 4771 tablet_bootstrap.cc:492] T 38d17bed4b8a466b9be4c0eee1d8f5df P 13dd5ff0fc59493a84afa5ed4f71c1b0: Bootstrap starting.
I20260928 20:19:30.453642 4248 catalog_manager.cc:4841] TS 13dd5ff0fc59493a84afa5ed4f71c1b0 (127.3.18.2:33777): tablet b606ad67275942289baa32313068d483 (table test-table [id=d72854ec94a545aea1e37cdf1d48ae50]) successfully deleted
I20260928 20:19:30.448269 4400 tablet_service.cc:1467] Processing CreateTablet for tablet 38d17bed4b8a466b9be4c0eee1d8f5df (DEFAULT_TABLE table=test-table [id=fb5b21539dff412fbd81253697c3b220]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:19:30.453786 4400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38d17bed4b8a466b9be4c0eee1d8f5df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:30.454205 4771 tablet_bootstrap.cc:654] T 38d17bed4b8a466b9be4c0eee1d8f5df P 13dd5ff0fc59493a84afa5ed4f71c1b0: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:30.454861 4771 tablet_bootstrap.cc:492] T 38d17bed4b8a466b9be4c0eee1d8f5df P 13dd5ff0fc59493a84afa5ed4f71c1b0: No bootstrap required, opened a new log
I20260928 20:19:30.454926 4771 ts_tablet_manager.cc:1397] T 38d17bed4b8a466b9be4c0eee1d8f5df P 13dd5ff0fc59493a84afa5ed4f71c1b0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 20:19:30.455080 4771 raft_consensus.cc:348] T 38d17bed4b8a466b9be4c0eee1d8f5df P 13dd5ff0fc59493a84afa5ed4f71c1b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3734cff4fd64fce9d89707d7d6e9e94" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40669 } } peers { permanent_uuid: "13dd5ff0fc59493a84afa5ed4f71c1b0" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 33777 } } peers { permanent_uuid: "8b1a6b51b8344a45a8f9918d5130860e" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 46789 } }
I20260928 20:19:30.455173 4771 raft_consensus.cc:374] T 38d17bed4b8a466b9be4c0eee1d8f5df P 13dd5ff0fc59493a84afa5ed4f71c1b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:30.455195 4771 raft_consensus.cc:729] T 38d17bed4b8a466b9be4c0eee1d8f5df P 13dd5ff0fc59493a84afa5ed4f71c1b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13dd5ff0fc59493a84afa5ed4f71c1b0, State: Initialized, Role: FOLLOWER
I20260928 20:19:30.455240 4771 consensus_queue.cc:260] T 38d17bed4b8a466b9be4c0eee1d8f5df P 13dd5ff0fc59493a84afa5ed4f71c1b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3734cff4fd64fce9d89707d7d6e9e94" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40669 } } peers { permanent_uuid: "13dd5ff0fc59493a84afa5ed4f71c1b0" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 33777 } } peers { permanent_uuid: "8b1a6b51b8344a45a8f9918d5130860e" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 46789 } }
I20260928 20:19:30.455346 4771 ts_tablet_manager.cc:1428] T 38d17bed4b8a466b9be4c0eee1d8f5df P 13dd5ff0fc59493a84afa5ed4f71c1b0: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 20:19:30.455499 4770 tablet_bootstrap.cc:492] T 38d17bed4b8a466b9be4c0eee1d8f5df P b3734cff4fd64fce9d89707d7d6e9e94: Bootstrap starting.
I20260928 20:19:30.456030 4770 tablet_bootstrap.cc:654] T 38d17bed4b8a466b9be4c0eee1d8f5df P b3734cff4fd64fce9d89707d7d6e9e94: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:30.447888 4793 ts_tablet_manager.cc:1905] T b606ad67275942289baa32313068d483 P b3734cff4fd64fce9d89707d7d6e9e94: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:19:30.456746 4770 tablet_bootstrap.cc:492] T 38d17bed4b8a466b9be4c0eee1d8f5df P b3734cff4fd64fce9d89707d7d6e9e94: No bootstrap required, opened a new log
I20260928 20:19:30.456808 4770 ts_tablet_manager.cc:1397] T 38d17bed4b8a466b9be4c0eee1d8f5df P b3734cff4fd64fce9d89707d7d6e9e94: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 20:19:30.456983 4770 raft_consensus.cc:348] T 38d17bed4b8a466b9be4c0eee1d8f5df P b3734cff4fd64fce9d89707d7d6e9e94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3734cff4fd64fce9d89707d7d6e9e94" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40669 } } peers { permanent_uuid: "13dd5ff0fc59493a84afa5ed4f71c1b0" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 33777 } } peers { permanent_uuid: "8b1a6b51b8344a45a8f9918d5130860e" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 46789 } }
I20260928 20:19:30.457060 4770 raft_consensus.cc:374] T 38d17bed4b8a466b9be4c0eee1d8f5df P b3734cff4fd64fce9d89707d7d6e9e94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:30.457080 4770 raft_consensus.cc:729] T 38d17bed4b8a466b9be4c0eee1d8f5df P b3734cff4fd64fce9d89707d7d6e9e94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3734cff4fd64fce9d89707d7d6e9e94, State: Initialized, Role: FOLLOWER
I20260928 20:19:30.457125 4770 consensus_queue.cc:260] T 38d17bed4b8a466b9be4c0eee1d8f5df P b3734cff4fd64fce9d89707d7d6e9e94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3734cff4fd64fce9d89707d7d6e9e94" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40669 } } peers { permanent_uuid: "13dd5ff0fc59493a84afa5ed4f71c1b0" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 33777 } } peers { permanent_uuid: "8b1a6b51b8344a45a8f9918d5130860e" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 46789 } }
I20260928 20:19:30.457348 4770 ts_tablet_manager.cc:1428] T 38d17bed4b8a466b9be4c0eee1d8f5df P b3734cff4fd64fce9d89707d7d6e9e94: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:30.462810 4793 ts_tablet_manager.cc:1918] T b606ad67275942289baa32313068d483 P b3734cff4fd64fce9d89707d7d6e9e94: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:19:30.462947 4793 log.cc:1196] T b606ad67275942289baa32313068d483 P b3734cff4fd64fce9d89707d7d6e9e94: Deleting WAL directory at /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/ts-0/wal/wals/b606ad67275942289baa32313068d483
I20260928 20:19:30.463541 4769 tablet_bootstrap.cc:492] T 38d17bed4b8a466b9be4c0eee1d8f5df P 8b1a6b51b8344a45a8f9918d5130860e: No bootstrap required, opened a new log
I20260928 20:19:30.463616 4769 ts_tablet_manager.cc:1397] T 38d17bed4b8a466b9be4c0eee1d8f5df P 8b1a6b51b8344a45a8f9918d5130860e: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.000s
I20260928 20:19:30.463850 4769 raft_consensus.cc:348] T 38d17bed4b8a466b9be4c0eee1d8f5df P 8b1a6b51b8344a45a8f9918d5130860e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3734cff4fd64fce9d89707d7d6e9e94" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40669 } } peers { permanent_uuid: "13dd5ff0fc59493a84afa5ed4f71c1b0" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 33777 } } peers { permanent_uuid: "8b1a6b51b8344a45a8f9918d5130860e" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 46789 } }
I20260928 20:19:30.464010 4769 raft_consensus.cc:374] T 38d17bed4b8a466b9be4c0eee1d8f5df P 8b1a6b51b8344a45a8f9918d5130860e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:30.464035 4769 raft_consensus.cc:729] T 38d17bed4b8a466b9be4c0eee1d8f5df P 8b1a6b51b8344a45a8f9918d5130860e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b1a6b51b8344a45a8f9918d5130860e, State: Initialized, Role: FOLLOWER
I20260928 20:19:30.464092 4769 consensus_queue.cc:260] T 38d17bed4b8a466b9be4c0eee1d8f5df P 8b1a6b51b8344a45a8f9918d5130860e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3734cff4fd64fce9d89707d7d6e9e94" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40669 } } peers { permanent_uuid: "13dd5ff0fc59493a84afa5ed4f71c1b0" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 33777 } } peers { permanent_uuid: "8b1a6b51b8344a45a8f9918d5130860e" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 46789 } }
I20260928 20:19:30.464187 4769 ts_tablet_manager.cc:1428] T 38d17bed4b8a466b9be4c0eee1d8f5df P 8b1a6b51b8344a45a8f9918d5130860e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 20:19:30.464560 4793 ts_tablet_manager.cc:1939] T b606ad67275942289baa32313068d483 P b3734cff4fd64fce9d89707d7d6e9e94: Deleting consensus metadata
I20260928 20:19:30.465186 4250 catalog_manager.cc:4841] TS b3734cff4fd64fce9d89707d7d6e9e94 (127.3.18.1:40669): tablet b606ad67275942289baa32313068d483 (table test-table [id=d72854ec94a545aea1e37cdf1d48ae50]) successfully deleted
W20260928 20:19:30.493142 4467 tablet.cc:2369] T 38d17bed4b8a466b9be4c0eee1d8f5df P b3734cff4fd64fce9d89707d7d6e9e94: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 20:19:30.515654 4775 raft_consensus.cc:482] T 38d17bed4b8a466b9be4c0eee1d8f5df P 8b1a6b51b8344a45a8f9918d5130860e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:19:30.515854 4775 raft_consensus.cc:504] T 38d17bed4b8a466b9be4c0eee1d8f5df P 8b1a6b51b8344a45a8f9918d5130860e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3734cff4fd64fce9d89707d7d6e9e94" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40669 } } peers { permanent_uuid: "13dd5ff0fc59493a84afa5ed4f71c1b0" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 33777 } } peers { permanent_uuid: "8b1a6b51b8344a45a8f9918d5130860e" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 46789 } }
I20260928 20:19:30.516327 4775 leader_election.cc:288] T 38d17bed4b8a466b9be4c0eee1d8f5df P 8b1a6b51b8344a45a8f9918d5130860e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3734cff4fd64fce9d89707d7d6e9e94 (127.3.18.1:40669), 13dd5ff0fc59493a84afa5ed4f71c1b0 (127.3.18.2:33777)
I20260928 20:19:30.519896 4798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:30.516514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.1:40669 (local address 127.3.18.3:49365)
0928 20:19:30.516789 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:30.516799 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:30.516880 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:30.517080 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:30.517080 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:30.517082 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:19:30.517158 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:30.517159 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.518016 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.518018 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:30.518835 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:30.518838 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.519570 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.519571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:30.519652 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:30.519657 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 20:19:30.519718 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 20:19:30.519788 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":142,"thread_start_us":69,"threads_started":1}
I20260928 20:19:30.519897 4768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:30.516579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:49365 (local address 127.3.18.1:40669)
0928 20:19:30.516631 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:30.516633 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:30.516852 (+ 219us) server_negotiation.cc:407] Connection header received
0928 20:19:30.516993 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:30.516994 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:30.517007 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:30.517061 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:19:30.517239 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.517913 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.518942 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.519463 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.519547 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:30.519559 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:30.519744 (+ 185us) server_negotiation.cc:299] Negotiation successful
0928 20:19:30.519819 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 20:19:30.520107 4639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:30.516513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.2:33777 (local address 127.3.18.3:45279)
0928 20:19:30.516561 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:30.516573 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:30.516629 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:30.516878 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:30.516880 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:30.516882 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:19:30.516951 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:30.516953 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.517890 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.517891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:30.519044 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:30.519047 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.519830 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.519831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:30.519919 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:30.519923 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:19:30.519995 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 20:19:30.520043 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 20:19:30.520479 4766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:30.516672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:45279 (local address 127.3.18.2:33777)
0928 20:19:30.516730 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:30.516733 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:30.516742 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:19:30.516775 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:30.516776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:30.516787 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:30.516865 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:19:30.517017 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.517780 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.519142 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.519668 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.519753 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:30.519763 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:30.520346 (+ 583us) server_negotiation.cc:299] Negotiation successful
0928 20:19:30.520399 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 20:19:30.520466 4420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38d17bed4b8a466b9be4c0eee1d8f5df" candidate_uuid: "8b1a6b51b8344a45a8f9918d5130860e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3734cff4fd64fce9d89707d7d6e9e94" is_pre_election: true
I20260928 20:19:30.520618 4420 raft_consensus.cc:2454] T 38d17bed4b8a466b9be4c0eee1d8f5df P b3734cff4fd64fce9d89707d7d6e9e94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b1a6b51b8344a45a8f9918d5130860e in term 0.
I20260928 20:19:30.520726 4560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38d17bed4b8a466b9be4c0eee1d8f5df" candidate_uuid: "8b1a6b51b8344a45a8f9918d5130860e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13dd5ff0fc59493a84afa5ed4f71c1b0" is_pre_election: true
I20260928 20:19:30.520807 4560 raft_consensus.cc:2454] T 38d17bed4b8a466b9be4c0eee1d8f5df P 13dd5ff0fc59493a84afa5ed4f71c1b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b1a6b51b8344a45a8f9918d5130860e in term 0.
I20260928 20:19:30.520823 4634 leader_election.cc:302] T 38d17bed4b8a466b9be4c0eee1d8f5df P 8b1a6b51b8344a45a8f9918d5130860e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b1a6b51b8344a45a8f9918d5130860e, b3734cff4fd64fce9d89707d7d6e9e94; no voters:
I20260928 20:19:30.520951 4775 raft_consensus.cc:2781] T 38d17bed4b8a466b9be4c0eee1d8f5df P 8b1a6b51b8344a45a8f9918d5130860e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 20:19:30.520982 4775 raft_consensus.cc:482] T 38d17bed4b8a466b9be4c0eee1d8f5df P 8b1a6b51b8344a45a8f9918d5130860e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 20:19:30.521001 4775 raft_consensus.cc:3037] T 38d17bed4b8a466b9be4c0eee1d8f5df P 8b1a6b51b8344a45a8f9918d5130860e [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:30.521908 4775 raft_consensus.cc:504] T 38d17bed4b8a466b9be4c0eee1d8f5df P 8b1a6b51b8344a45a8f9918d5130860e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3734cff4fd64fce9d89707d7d6e9e94" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40669 } } peers { permanent_uuid: "13dd5ff0fc59493a84afa5ed4f71c1b0" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 33777 } } peers { permanent_uuid: "8b1a6b51b8344a45a8f9918d5130860e" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 46789 } }
I20260928 20:19:30.522061 4775 leader_election.cc:288] T 38d17bed4b8a466b9be4c0eee1d8f5df P 8b1a6b51b8344a45a8f9918d5130860e [CANDIDATE]: Term 1 election: Requested vote from peers b3734cff4fd64fce9d89707d7d6e9e94 (127.3.18.1:40669), 13dd5ff0fc59493a84afa5ed4f71c1b0 (127.3.18.2:33777)
I20260928 20:19:30.522279 4420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38d17bed4b8a466b9be4c0eee1d8f5df" candidate_uuid: "8b1a6b51b8344a45a8f9918d5130860e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3734cff4fd64fce9d89707d7d6e9e94"
I20260928 20:19:30.522408 4560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38d17bed4b8a466b9be4c0eee1d8f5df" candidate_uuid: "8b1a6b51b8344a45a8f9918d5130860e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13dd5ff0fc59493a84afa5ed4f71c1b0"
I20260928 20:19:30.522500 4560 raft_consensus.cc:3037] T 38d17bed4b8a466b9be4c0eee1d8f5df P 13dd5ff0fc59493a84afa5ed4f71c1b0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:30.522403 4420 raft_consensus.cc:3037] T 38d17bed4b8a466b9be4c0eee1d8f5df P b3734cff4fd64fce9d89707d7d6e9e94 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:30.523262 4420 raft_consensus.cc:2454] T 38d17bed4b8a466b9be4c0eee1d8f5df P b3734cff4fd64fce9d89707d7d6e9e94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b1a6b51b8344a45a8f9918d5130860e in term 1.
I20260928 20:19:30.523279 4560 raft_consensus.cc:2454] T 38d17bed4b8a466b9be4c0eee1d8f5df P 13dd5ff0fc59493a84afa5ed4f71c1b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b1a6b51b8344a45a8f9918d5130860e in term 1.
I20260928 20:19:30.523494 4632 leader_election.cc:302] T 38d17bed4b8a466b9be4c0eee1d8f5df P 8b1a6b51b8344a45a8f9918d5130860e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13dd5ff0fc59493a84afa5ed4f71c1b0, 8b1a6b51b8344a45a8f9918d5130860e; no voters:
I20260928 20:19:30.523612 4775 raft_consensus.cc:2781] T 38d17bed4b8a466b9be4c0eee1d8f5df P 8b1a6b51b8344a45a8f9918d5130860e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:19:30.523787 4775 raft_consensus.cc:686] T 38d17bed4b8a466b9be4c0eee1d8f5df P 8b1a6b51b8344a45a8f9918d5130860e [term 1 LEADER]: Becoming Leader. State: Replica: 8b1a6b51b8344a45a8f9918d5130860e, State: Running, Role: LEADER
I20260928 20:19:30.523872 4775 consensus_queue.cc:237] T 38d17bed4b8a466b9be4c0eee1d8f5df P 8b1a6b51b8344a45a8f9918d5130860e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3734cff4fd64fce9d89707d7d6e9e94" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40669 } } peers { permanent_uuid: "13dd5ff0fc59493a84afa5ed4f71c1b0" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 33777 } } peers { permanent_uuid: "8b1a6b51b8344a45a8f9918d5130860e" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 46789 } }
I20260928 20:19:30.524828 4261 catalog_manager.cc:5495] T 38d17bed4b8a466b9be4c0eee1d8f5df P 8b1a6b51b8344a45a8f9918d5130860e reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b1a6b51b8344a45a8f9918d5130860e (127.3.18.3). New cstate: current_term: 1 leader_uuid: "8b1a6b51b8344a45a8f9918d5130860e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3734cff4fd64fce9d89707d7d6e9e94" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13dd5ff0fc59493a84afa5ed4f71c1b0" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 33777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b1a6b51b8344a45a8f9918d5130860e" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 46789 } health_report { overall_health: HEALTHY } } }
Sep 28 20:19:30 dist-test-slave-b83v krb5kdc[4218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626769, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.62@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:19:30.572860 4312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:30.564795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51796 (local address 127.3.18.62:40329)
0928 20:19:30.564870 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:30.564873 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:30.565030 (+ 157us) server_negotiation.cc:407] Connection header received
0928 20:19:30.565097 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:30.565098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:30.565111 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:30.565167 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:30.565698 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.566420 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.567236 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.567526 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.567616 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:30.570055 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:30.570058 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:30.570058 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:30.570067 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:30.571569 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:30.572077 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:30.572078 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:30.572078 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:30.572136 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:30.572351 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:30.572352 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:30.572352 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:30.572479 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:30.572532 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:30.572714 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 20:19:30.572794 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 20:19:30.573053 4806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:30.564735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40329 (local address 127.0.0.1:51796)
0928 20:19:30.564942 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:30.564952 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:30.565024 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:30.565191 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:30.565192 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:30.565545 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:30.565601 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:30.565604 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.566532 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.566533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:30.567088 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:30.567090 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.567615 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.567616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:30.567672 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:30.568313 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:19:30.568318 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:30.569913 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:30.571678 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:30.571687 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:30.571690 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:30.571987 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:30.572213 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:30.572214 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:30.572214 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:30.572286 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:30.572539 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:30.572547 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:30.572615 (+ 68us) client_negotiation.cc:769] Sending connection context
0928 20:19:30.572693 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 20:19:30.572879 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":61,"thread_start_us":115,"threads_started":1}
I20260928 20:19:30.575241 4261 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:51796:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 20:19:30.575505 4261 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 20:19:30.576956 4400 tablet_service.cc:1467] Processing CreateTablet for tablet fb8c1b9a08c04a7faed3f4cb08beb078 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9e4b6d16848b45d5b0c74108417ef103]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 20:19:30.577108 4400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb8c1b9a08c04a7faed3f4cb08beb078. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:30.578289 4770 tablet_bootstrap.cc:492] T fb8c1b9a08c04a7faed3f4cb08beb078 P b3734cff4fd64fce9d89707d7d6e9e94: Bootstrap starting.
I20260928 20:19:30.578413 4680 tablet_service.cc:1467] Processing CreateTablet for tablet fb8c1b9a08c04a7faed3f4cb08beb078 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9e4b6d16848b45d5b0c74108417ef103]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 20:19:30.578550 4680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb8c1b9a08c04a7faed3f4cb08beb078. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:30.578645 4770 tablet_bootstrap.cc:654] T fb8c1b9a08c04a7faed3f4cb08beb078 P b3734cff4fd64fce9d89707d7d6e9e94: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:30.579833 4769 tablet_bootstrap.cc:492] T fb8c1b9a08c04a7faed3f4cb08beb078 P 8b1a6b51b8344a45a8f9918d5130860e: Bootstrap starting.
I20260928 20:19:30.579977 4540 tablet_service.cc:1467] Processing CreateTablet for tablet fb8c1b9a08c04a7faed3f4cb08beb078 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9e4b6d16848b45d5b0c74108417ef103]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 20:19:30.580148 4540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb8c1b9a08c04a7faed3f4cb08beb078. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:30.580329 4769 tablet_bootstrap.cc:654] T fb8c1b9a08c04a7faed3f4cb08beb078 P 8b1a6b51b8344a45a8f9918d5130860e: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:30.581097 4769 tablet_bootstrap.cc:492] T fb8c1b9a08c04a7faed3f4cb08beb078 P 8b1a6b51b8344a45a8f9918d5130860e: No bootstrap required, opened a new log
I20260928 20:19:30.581148 4769 ts_tablet_manager.cc:1397] T fb8c1b9a08c04a7faed3f4cb08beb078 P 8b1a6b51b8344a45a8f9918d5130860e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 20:19:30.581453 4769 raft_consensus.cc:348] T fb8c1b9a08c04a7faed3f4cb08beb078 P 8b1a6b51b8344a45a8f9918d5130860e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3734cff4fd64fce9d89707d7d6e9e94" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40669 } } peers { permanent_uuid: "13dd5ff0fc59493a84afa5ed4f71c1b0" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 33777 } } peers { permanent_uuid: "8b1a6b51b8344a45a8f9918d5130860e" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 46789 } }
I20260928 20:19:30.581534 4769 raft_consensus.cc:374] T fb8c1b9a08c04a7faed3f4cb08beb078 P 8b1a6b51b8344a45a8f9918d5130860e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:30.581552 4769 raft_consensus.cc:729] T fb8c1b9a08c04a7faed3f4cb08beb078 P 8b1a6b51b8344a45a8f9918d5130860e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b1a6b51b8344a45a8f9918d5130860e, State: Initialized, Role: FOLLOWER
I20260928 20:19:30.581631 4769 consensus_queue.cc:260] T fb8c1b9a08c04a7faed3f4cb08beb078 P 8b1a6b51b8344a45a8f9918d5130860e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3734cff4fd64fce9d89707d7d6e9e94" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40669 } } peers { permanent_uuid: "13dd5ff0fc59493a84afa5ed4f71c1b0" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 33777 } } peers { permanent_uuid: "8b1a6b51b8344a45a8f9918d5130860e" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 46789 } }
I20260928 20:19:30.581722 4769 ts_tablet_manager.cc:1428] T fb8c1b9a08c04a7faed3f4cb08beb078 P 8b1a6b51b8344a45a8f9918d5130860e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 20:19:30.581831 4771 tablet_bootstrap.cc:492] T fb8c1b9a08c04a7faed3f4cb08beb078 P 13dd5ff0fc59493a84afa5ed4f71c1b0: Bootstrap starting.
I20260928 20:19:30.582094 4770 tablet_bootstrap.cc:492] T fb8c1b9a08c04a7faed3f4cb08beb078 P b3734cff4fd64fce9d89707d7d6e9e94: No bootstrap required, opened a new log
I20260928 20:19:30.582144 4770 ts_tablet_manager.cc:1397] T fb8c1b9a08c04a7faed3f4cb08beb078 P b3734cff4fd64fce9d89707d7d6e9e94: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260928 20:19:30.582213 4771 tablet_bootstrap.cc:654] T fb8c1b9a08c04a7faed3f4cb08beb078 P 13dd5ff0fc59493a84afa5ed4f71c1b0: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:30.582268 4770 raft_consensus.cc:348] T fb8c1b9a08c04a7faed3f4cb08beb078 P b3734cff4fd64fce9d89707d7d6e9e94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3734cff4fd64fce9d89707d7d6e9e94" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40669 } } peers { permanent_uuid: "13dd5ff0fc59493a84afa5ed4f71c1b0" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 33777 } } peers { permanent_uuid: "8b1a6b51b8344a45a8f9918d5130860e" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 46789 } }
I20260928 20:19:30.582345 4770 raft_consensus.cc:374] T fb8c1b9a08c04a7faed3f4cb08beb078 P b3734cff4fd64fce9d89707d7d6e9e94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:30.582363 4770 raft_consensus.cc:729] T fb8c1b9a08c04a7faed3f4cb08beb078 P b3734cff4fd64fce9d89707d7d6e9e94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3734cff4fd64fce9d89707d7d6e9e94, State: Initialized, Role: FOLLOWER
I20260928 20:19:30.582409 4770 consensus_queue.cc:260] T fb8c1b9a08c04a7faed3f4cb08beb078 P b3734cff4fd64fce9d89707d7d6e9e94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3734cff4fd64fce9d89707d7d6e9e94" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40669 } } peers { permanent_uuid: "13dd5ff0fc59493a84afa5ed4f71c1b0" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 33777 } } peers { permanent_uuid: "8b1a6b51b8344a45a8f9918d5130860e" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 46789 } }
I20260928 20:19:30.582505 4770 ts_tablet_manager.cc:1428] T fb8c1b9a08c04a7faed3f4cb08beb078 P b3734cff4fd64fce9d89707d7d6e9e94: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:30.582573 4775 tablet_replica.cc:439] T fb8c1b9a08c04a7faed3f4cb08beb078 P 8b1a6b51b8344a45a8f9918d5130860e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3734cff4fd64fce9d89707d7d6e9e94" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40669 } } peers { permanent_uuid: "13dd5ff0fc59493a84afa5ed4f71c1b0" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 33777 } } peers { permanent_uuid: "8b1a6b51b8344a45a8f9918d5130860e" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 46789 } } }
I20260928 20:19:30.582655 4775 tablet_replica.cc:442] T fb8c1b9a08c04a7faed3f4cb08beb078 P 8b1a6b51b8344a45a8f9918d5130860e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 20:19:30.582753 4789 tablet_replica.cc:439] T fb8c1b9a08c04a7faed3f4cb08beb078 P b3734cff4fd64fce9d89707d7d6e9e94: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3734cff4fd64fce9d89707d7d6e9e94" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40669 } } peers { permanent_uuid: "13dd5ff0fc59493a84afa5ed4f71c1b0" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 33777 } } peers { permanent_uuid: "8b1a6b51b8344a45a8f9918d5130860e" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 46789 } } }
I20260928 20:19:30.582851 4789 tablet_replica.cc:442] T fb8c1b9a08c04a7faed3f4cb08beb078 P b3734cff4fd64fce9d89707d7d6e9e94: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 20:19:30.583544 4771 tablet_bootstrap.cc:492] T fb8c1b9a08c04a7faed3f4cb08beb078 P 13dd5ff0fc59493a84afa5ed4f71c1b0: No bootstrap required, opened a new log
I20260928 20:19:30.583590 4771 ts_tablet_manager.cc:1397] T fb8c1b9a08c04a7faed3f4cb08beb078 P 13dd5ff0fc59493a84afa5ed4f71c1b0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 20:19:30.583806 4771 raft_consensus.cc:348] T fb8c1b9a08c04a7faed3f4cb08beb078 P 13dd5ff0fc59493a84afa5ed4f71c1b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3734cff4fd64fce9d89707d7d6e9e94" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40669 } } peers { permanent_uuid: "13dd5ff0fc59493a84afa5ed4f71c1b0" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 33777 } } peers { permanent_uuid: "8b1a6b51b8344a45a8f9918d5130860e" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 46789 } }
I20260928 20:19:30.583880 4771 raft_consensus.cc:374] T fb8c1b9a08c04a7faed3f4cb08beb078 P 13dd5ff0fc59493a84afa5ed4f71c1b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:30.583895 4771 raft_consensus.cc:729] T fb8c1b9a08c04a7faed3f4cb08beb078 P 13dd5ff0fc59493a84afa5ed4f71c1b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13dd5ff0fc59493a84afa5ed4f71c1b0, State: Initialized, Role: FOLLOWER
I20260928 20:19:30.583956 4771 consensus_queue.cc:260] T fb8c1b9a08c04a7faed3f4cb08beb078 P 13dd5ff0fc59493a84afa5ed4f71c1b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3734cff4fd64fce9d89707d7d6e9e94" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40669 } } peers { permanent_uuid: "13dd5ff0fc59493a84afa5ed4f71c1b0" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 33777 } } peers { permanent_uuid: "8b1a6b51b8344a45a8f9918d5130860e" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 46789 } }
I20260928 20:19:30.584051 4771 ts_tablet_manager.cc:1428] T fb8c1b9a08c04a7faed3f4cb08beb078 P 13dd5ff0fc59493a84afa5ed4f71c1b0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:30.584127 4777 tablet_replica.cc:439] T fb8c1b9a08c04a7faed3f4cb08beb078 P 13dd5ff0fc59493a84afa5ed4f71c1b0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3734cff4fd64fce9d89707d7d6e9e94" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40669 } } peers { permanent_uuid: "13dd5ff0fc59493a84afa5ed4f71c1b0" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 33777 } } peers { permanent_uuid: "8b1a6b51b8344a45a8f9918d5130860e" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 46789 } } }
I20260928 20:19:30.584208 4777 tablet_replica.cc:442] T fb8c1b9a08c04a7faed3f4cb08beb078 P 13dd5ff0fc59493a84afa5ed4f71c1b0: This TxnStatusTablet replica's current role is: FOLLOWER
W20260928 20:19:30.606660 4747 tablet.cc:2369] T 38d17bed4b8a466b9be4c0eee1d8f5df P 8b1a6b51b8344a45a8f9918d5130860e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 20:19:30.620363 4777 raft_consensus.cc:482] T fb8c1b9a08c04a7faed3f4cb08beb078 P 13dd5ff0fc59493a84afa5ed4f71c1b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:19:30.620477 4777 raft_consensus.cc:504] T fb8c1b9a08c04a7faed3f4cb08beb078 P 13dd5ff0fc59493a84afa5ed4f71c1b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3734cff4fd64fce9d89707d7d6e9e94" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40669 } } peers { permanent_uuid: "13dd5ff0fc59493a84afa5ed4f71c1b0" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 33777 } } peers { permanent_uuid: "8b1a6b51b8344a45a8f9918d5130860e" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 46789 } }
I20260928 20:19:30.620941 4777 leader_election.cc:288] T fb8c1b9a08c04a7faed3f4cb08beb078 P 13dd5ff0fc59493a84afa5ed4f71c1b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3734cff4fd64fce9d89707d7d6e9e94 (127.3.18.1:40669), 8b1a6b51b8344a45a8f9918d5130860e (127.3.18.3:46789)
I20260928 20:19:30.624513 4499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:30.621178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.1:40669 (local address 127.3.18.2:56827)
0928 20:19:30.621274 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:30.621318 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:30.621410 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:30.621591 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:30.621592 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:30.621594 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:19:30.621693 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:30.621696 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.622477 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.622478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:30.623403 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:30.623406 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.624289 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.624290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:30.624366 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:30.624370 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:19:30.624420 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 20:19:30.624453 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 20:19:30.624576 4768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:30.621286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:56827 (local address 127.3.18.1:40669)
0928 20:19:30.621341 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:30.621343 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:30.621379 (+ 36us) server_negotiation.cc:407] Connection header received
0928 20:19:30.621491 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:30.621493 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:30.621510 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:30.621572 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:19:30.621787 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.622366 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.623490 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.624202 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.624283 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:30.624301 (+ 18us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:30.624445 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 20:19:30.624487 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 20:19:30.624908 4420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8c1b9a08c04a7faed3f4cb08beb078" candidate_uuid: "13dd5ff0fc59493a84afa5ed4f71c1b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3734cff4fd64fce9d89707d7d6e9e94" is_pre_election: true
I20260928 20:19:30.625005 4420 raft_consensus.cc:2454] T fb8c1b9a08c04a7faed3f4cb08beb078 P b3734cff4fd64fce9d89707d7d6e9e94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13dd5ff0fc59493a84afa5ed4f71c1b0 in term 0.
I20260928 20:19:30.625160 4494 leader_election.cc:302] T fb8c1b9a08c04a7faed3f4cb08beb078 P 13dd5ff0fc59493a84afa5ed4f71c1b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13dd5ff0fc59493a84afa5ed4f71c1b0, b3734cff4fd64fce9d89707d7d6e9e94; no voters:
I20260928 20:19:30.625290 4777 raft_consensus.cc:2781] T fb8c1b9a08c04a7faed3f4cb08beb078 P 13dd5ff0fc59493a84afa5ed4f71c1b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 20:19:30.625334 4777 raft_consensus.cc:482] T fb8c1b9a08c04a7faed3f4cb08beb078 P 13dd5ff0fc59493a84afa5ed4f71c1b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 20:19:30.625367 4777 raft_consensus.cc:3037] T fb8c1b9a08c04a7faed3f4cb08beb078 P 13dd5ff0fc59493a84afa5ed4f71c1b0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:30.626137 4765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:30.621415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:51819 (local address 127.3.18.3:46789)
0928 20:19:30.621463 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:30.621465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:30.621574 (+ 109us) server_negotiation.cc:407] Connection header received
0928 20:19:30.621627 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:30.621628 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:30.621637 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:30.621705 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:19:30.621929 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.622454 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.625363 (+ 2909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.625774 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.625834 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:30.625842 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:30.626055 (+ 213us) server_negotiation.cc:299] Negotiation successful
0928 20:19:30.626087 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 20:19:30.626180 4810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:30.621298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.3:46789 (local address 127.3.18.2:51819)
0928 20:19:30.621486 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:30.621497 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:30.621569 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:30.621706 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:30.621706 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:30.621707 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:19:30.621809 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:30.621811 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.622537 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.622537 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:30.625275 (+ 2738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:30.625278 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.625861 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.625862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:30.625968 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:30.625976 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 20:19:30.626033 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 20:19:30.626081 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":123,"threads_started":1}
I20260928 20:19:30.626238 4777 raft_consensus.cc:504] T fb8c1b9a08c04a7faed3f4cb08beb078 P 13dd5ff0fc59493a84afa5ed4f71c1b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3734cff4fd64fce9d89707d7d6e9e94" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40669 } } peers { permanent_uuid: "13dd5ff0fc59493a84afa5ed4f71c1b0" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 33777 } } peers { permanent_uuid: "8b1a6b51b8344a45a8f9918d5130860e" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 46789 } }
I20260928 20:19:30.626351 4777 leader_election.cc:288] T fb8c1b9a08c04a7faed3f4cb08beb078 P 13dd5ff0fc59493a84afa5ed4f71c1b0 [CANDIDATE]: Term 1 election: Requested vote from peers b3734cff4fd64fce9d89707d7d6e9e94 (127.3.18.1:40669), 8b1a6b51b8344a45a8f9918d5130860e (127.3.18.3:46789)
I20260928 20:19:30.626550 4420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8c1b9a08c04a7faed3f4cb08beb078" candidate_uuid: "13dd5ff0fc59493a84afa5ed4f71c1b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3734cff4fd64fce9d89707d7d6e9e94"
I20260928 20:19:30.626664 4420 raft_consensus.cc:3037] T fb8c1b9a08c04a7faed3f4cb08beb078 P b3734cff4fd64fce9d89707d7d6e9e94 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:30.626595 4700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8c1b9a08c04a7faed3f4cb08beb078" candidate_uuid: "13dd5ff0fc59493a84afa5ed4f71c1b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b1a6b51b8344a45a8f9918d5130860e" is_pre_election: true
I20260928 20:19:30.626674 4699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8c1b9a08c04a7faed3f4cb08beb078" candidate_uuid: "13dd5ff0fc59493a84afa5ed4f71c1b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b1a6b51b8344a45a8f9918d5130860e"
I20260928 20:19:30.626727 4700 raft_consensus.cc:2454] T fb8c1b9a08c04a7faed3f4cb08beb078 P 8b1a6b51b8344a45a8f9918d5130860e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13dd5ff0fc59493a84afa5ed4f71c1b0 in term 0.
I20260928 20:19:30.627363 4420 raft_consensus.cc:2454] T fb8c1b9a08c04a7faed3f4cb08beb078 P b3734cff4fd64fce9d89707d7d6e9e94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13dd5ff0fc59493a84afa5ed4f71c1b0 in term 1.
I20260928 20:19:30.627497 4494 leader_election.cc:302] T fb8c1b9a08c04a7faed3f4cb08beb078 P 13dd5ff0fc59493a84afa5ed4f71c1b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 13dd5ff0fc59493a84afa5ed4f71c1b0, b3734cff4fd64fce9d89707d7d6e9e94; no voters: 8b1a6b51b8344a45a8f9918d5130860e
I20260928 20:19:30.627602 4777 raft_consensus.cc:2781] T fb8c1b9a08c04a7faed3f4cb08beb078 P 13dd5ff0fc59493a84afa5ed4f71c1b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:19:30.627810 4777 raft_consensus.cc:686] T fb8c1b9a08c04a7faed3f4cb08beb078 P 13dd5ff0fc59493a84afa5ed4f71c1b0 [term 1 LEADER]: Becoming Leader. State: Replica: 13dd5ff0fc59493a84afa5ed4f71c1b0, State: Running, Role: LEADER
I20260928 20:19:30.627889 4777 consensus_queue.cc:237] T fb8c1b9a08c04a7faed3f4cb08beb078 P 13dd5ff0fc59493a84afa5ed4f71c1b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3734cff4fd64fce9d89707d7d6e9e94" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40669 } } peers { permanent_uuid: "13dd5ff0fc59493a84afa5ed4f71c1b0" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 33777 } } peers { permanent_uuid: "8b1a6b51b8344a45a8f9918d5130860e" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 46789 } }
I20260928 20:19:30.628407 4811 tablet_replica.cc:439] T fb8c1b9a08c04a7faed3f4cb08beb078 P 13dd5ff0fc59493a84afa5ed4f71c1b0: TxnStatusTablet state changed. Reason: New leader 13dd5ff0fc59493a84afa5ed4f71c1b0. Latest consensus state: current_term: 1 leader_uuid: "13dd5ff0fc59493a84afa5ed4f71c1b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3734cff4fd64fce9d89707d7d6e9e94" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40669 } } peers { permanent_uuid: "13dd5ff0fc59493a84afa5ed4f71c1b0" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 33777 } } peers { permanent_uuid: "8b1a6b51b8344a45a8f9918d5130860e" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 46789 } } }
I20260928 20:19:30.628500 4811 tablet_replica.cc:442] T fb8c1b9a08c04a7faed3f4cb08beb078 P 13dd5ff0fc59493a84afa5ed4f71c1b0: This TxnStatusTablet replica's current role is: LEADER
I20260928 20:19:30.628686 4813 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 20:19:30.628713 4263 catalog_manager.cc:5495] T fb8c1b9a08c04a7faed3f4cb08beb078 P 13dd5ff0fc59493a84afa5ed4f71c1b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 13dd5ff0fc59493a84afa5ed4f71c1b0 (127.3.18.2). New cstate: current_term: 1 leader_uuid: "13dd5ff0fc59493a84afa5ed4f71c1b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3734cff4fd64fce9d89707d7d6e9e94" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13dd5ff0fc59493a84afa5ed4f71c1b0" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 33777 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b1a6b51b8344a45a8f9918d5130860e" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 46789 } health_report { overall_health: UNKNOWN } } }
I20260928 20:19:30.628820 4813 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 20:19:30.641124 4766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:30.638117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52248 (local address 127.3.18.2:33777)
0928 20:19:30.638168 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:30.638170 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:30.638246 (+ 76us) server_negotiation.cc:407] Connection header received
0928 20:19:30.638348 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:30.638348 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:30.638378 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:30.638436 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:19:30.638600 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.639074 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.640252 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.640550 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.640620 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:30.640810 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:19:30.640905 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:19:30.640962 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:30.641031 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 20:19:30.641053 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 20:19:30.641173 4806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:30.638043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.2:33777 (local address 127.0.0.1:52248)
0928 20:19:30.638193 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:30.638206 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:30.638269 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:30.638458 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:30.638459 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:30.638460 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 20:19:30.638513 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:30.638516 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.639147 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.639148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:30.640153 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:30.640156 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.640626 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.640627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:30.640701 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:30.640720 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 20:19:30.640966 (+ 246us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 20:19:30.640974 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 20:19:30.641019 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 20:19:30.641071 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105}
I20260928 20:19:30.642542 4700 raft_consensus.cc:3037] T fb8c1b9a08c04a7faed3f4cb08beb078 P 8b1a6b51b8344a45a8f9918d5130860e [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:30.642686 4420 raft_consensus.cc:1261] T fb8c1b9a08c04a7faed3f4cb08beb078 P b3734cff4fd64fce9d89707d7d6e9e94 [term 1 FOLLOWER]: Refusing update from remote peer 13dd5ff0fc59493a84afa5ed4f71c1b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:19:30.642963 4811 consensus_queue.cc:1035] T fb8c1b9a08c04a7faed3f4cb08beb078 P 13dd5ff0fc59493a84afa5ed4f71c1b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3734cff4fd64fce9d89707d7d6e9e94" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:19:30.643507 4700 raft_consensus.cc:1261] T fb8c1b9a08c04a7faed3f4cb08beb078 P 8b1a6b51b8344a45a8f9918d5130860e [term 1 FOLLOWER]: Refusing update from remote peer 13dd5ff0fc59493a84afa5ed4f71c1b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:19:30.643870 4789 tablet_replica.cc:439] T fb8c1b9a08c04a7faed3f4cb08beb078 P b3734cff4fd64fce9d89707d7d6e9e94: TxnStatusTablet state changed. Reason: New leader 13dd5ff0fc59493a84afa5ed4f71c1b0. Latest consensus state: current_term: 1 leader_uuid: "13dd5ff0fc59493a84afa5ed4f71c1b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3734cff4fd64fce9d89707d7d6e9e94" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40669 } } peers { permanent_uuid: "13dd5ff0fc59493a84afa5ed4f71c1b0" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 33777 } } peers { permanent_uuid: "8b1a6b51b8344a45a8f9918d5130860e" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 46789 } } }
I20260928 20:19:30.643992 4789 tablet_replica.cc:442] T fb8c1b9a08c04a7faed3f4cb08beb078 P b3734cff4fd64fce9d89707d7d6e9e94: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 20:19:30.644131 4811 consensus_queue.cc:1035] T fb8c1b9a08c04a7faed3f4cb08beb078 P 13dd5ff0fc59493a84afa5ed4f71c1b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b1a6b51b8344a45a8f9918d5130860e" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 46789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:19:30.644687 4811 tablet_replica.cc:439] T fb8c1b9a08c04a7faed3f4cb08beb078 P 13dd5ff0fc59493a84afa5ed4f71c1b0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "13dd5ff0fc59493a84afa5ed4f71c1b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3734cff4fd64fce9d89707d7d6e9e94" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40669 } } peers { permanent_uuid: "13dd5ff0fc59493a84afa5ed4f71c1b0" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 33777 } } peers { permanent_uuid: "8b1a6b51b8344a45a8f9918d5130860e" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 46789 } } }
I20260928 20:19:30.644769 4811 tablet_replica.cc:442] T fb8c1b9a08c04a7faed3f4cb08beb078 P 13dd5ff0fc59493a84afa5ed4f71c1b0: This TxnStatusTablet replica's current role is: LEADER
I20260928 20:19:30.644788 4775 tablet_replica.cc:439] T fb8c1b9a08c04a7faed3f4cb08beb078 P 8b1a6b51b8344a45a8f9918d5130860e: TxnStatusTablet state changed. Reason: New leader 13dd5ff0fc59493a84afa5ed4f71c1b0. Latest consensus state: current_term: 1 leader_uuid: "13dd5ff0fc59493a84afa5ed4f71c1b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3734cff4fd64fce9d89707d7d6e9e94" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40669 } } peers { permanent_uuid: "13dd5ff0fc59493a84afa5ed4f71c1b0" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 33777 } } peers { permanent_uuid: "8b1a6b51b8344a45a8f9918d5130860e" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 46789 } } }
I20260928 20:19:30.644870 4775 tablet_replica.cc:442] T fb8c1b9a08c04a7faed3f4cb08beb078 P 8b1a6b51b8344a45a8f9918d5130860e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 20:19:30.645793 4789 tablet_replica.cc:439] T fb8c1b9a08c04a7faed3f4cb08beb078 P b3734cff4fd64fce9d89707d7d6e9e94: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "13dd5ff0fc59493a84afa5ed4f71c1b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3734cff4fd64fce9d89707d7d6e9e94" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40669 } } peers { permanent_uuid: "13dd5ff0fc59493a84afa5ed4f71c1b0" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 33777 } } peers { permanent_uuid: "8b1a6b51b8344a45a8f9918d5130860e" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 46789 } } }
I20260928 20:19:30.645884 4789 tablet_replica.cc:442] T fb8c1b9a08c04a7faed3f4cb08beb078 P b3734cff4fd64fce9d89707d7d6e9e94: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 20:19:30.645993 4777 tablet_replica.cc:439] T fb8c1b9a08c04a7faed3f4cb08beb078 P 13dd5ff0fc59493a84afa5ed4f71c1b0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "13dd5ff0fc59493a84afa5ed4f71c1b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3734cff4fd64fce9d89707d7d6e9e94" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40669 } } peers { permanent_uuid: "13dd5ff0fc59493a84afa5ed4f71c1b0" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 33777 } } peers { permanent_uuid: "8b1a6b51b8344a45a8f9918d5130860e" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 46789 } } }
I20260928 20:19:30.646067 4777 tablet_replica.cc:442] T fb8c1b9a08c04a7faed3f4cb08beb078 P 13dd5ff0fc59493a84afa5ed4f71c1b0: This TxnStatusTablet replica's current role is: LEADER
I20260928 20:19:30.646427 4775 tablet_replica.cc:439] T fb8c1b9a08c04a7faed3f4cb08beb078 P 8b1a6b51b8344a45a8f9918d5130860e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "13dd5ff0fc59493a84afa5ed4f71c1b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3734cff4fd64fce9d89707d7d6e9e94" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40669 } } peers { permanent_uuid: "13dd5ff0fc59493a84afa5ed4f71c1b0" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 33777 } } peers { permanent_uuid: "8b1a6b51b8344a45a8f9918d5130860e" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 46789 } } }
I20260928 20:19:30.646507 4775 tablet_replica.cc:442] T fb8c1b9a08c04a7faed3f4cb08beb078 P 8b1a6b51b8344a45a8f9918d5130860e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 20:19:30.649832 4765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:30.646754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37624 (local address 127.3.18.3:46789)
0928 20:19:30.646790 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:30.646792 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:30.646801 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:19:30.646870 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:30.646871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:30.646890 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:30.646962 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:19:30.647154 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.647923 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.648668 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.648993 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.649075 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:30.649389 (+ 314us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:19:30.649523 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:19:30.649605 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:30.649724 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 20:19:30.649747 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 20:19:30.655335 4752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:30.652427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.2:33777 (local address 127.3.18.3:40503)
0928 20:19:30.652537 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:30.652551 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:30.652611 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:30.652771 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:30.652771 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:30.652773 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 20:19:30.652833 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:30.652834 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.653755 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.653756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:30.654399 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:30.654402 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.654879 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.654880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:30.654942 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:30.654958 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 20:19:30.655185 (+ 227us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 20:19:30.655187 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 20:19:30.655234 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 20:19:30.655277 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 20:19:30.655351 4766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:30.652582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:40503 (local address 127.3.18.2:33777)
0928 20:19:30.652659 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:30.652661 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:30.652671 (+ 10us) server_negotiation.cc:407] Connection header received
0928 20:19:30.652697 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:30.652698 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:30.652709 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:30.652766 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:19:30.652892 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.653642 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.654483 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.654786 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.654872 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:30.655042 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:19:30.655079 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:19:30.655142 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:30.655250 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 20:19:30.655272 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 20:19:30.658021 4560 raft_consensus.cc:1261] T 38d17bed4b8a466b9be4c0eee1d8f5df P 13dd5ff0fc59493a84afa5ed4f71c1b0 [term 1 FOLLOWER]: Refusing update from remote peer 8b1a6b51b8344a45a8f9918d5130860e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:19:30.658030 4420 raft_consensus.cc:1261] T 38d17bed4b8a466b9be4c0eee1d8f5df P b3734cff4fd64fce9d89707d7d6e9e94 [term 1 FOLLOWER]: Refusing update from remote peer 8b1a6b51b8344a45a8f9918d5130860e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:19:30.658332 4775 consensus_queue.cc:1035] T 38d17bed4b8a466b9be4c0eee1d8f5df P 8b1a6b51b8344a45a8f9918d5130860e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3734cff4fd64fce9d89707d7d6e9e94" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:19:30.658438 4799 consensus_queue.cc:1035] T 38d17bed4b8a466b9be4c0eee1d8f5df P 8b1a6b51b8344a45a8f9918d5130860e [LEADER]: Connected to new peer: Peer: permanent_uuid: "13dd5ff0fc59493a84afa5ed4f71c1b0" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 33777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:19:30.667418 4765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:30.664784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:54047 (local address 127.3.18.3:46789)
0928 20:19:30.664927 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:30.664929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:30.664937 (+ 8us) server_negotiation.cc:407] Connection header received
0928 20:19:30.664967 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:30.664967 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:30.664975 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:30.665021 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:19:30.665174 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.665619 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.666731 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.666933 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.666983 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:30.667191 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:19:30.667222 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:19:30.667275 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:30.667347 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 20:19:30.667365 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260928 20:19:30.667486 4612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:30.664636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.3:46789 (local address 127.3.18.2:54047)
0928 20:19:30.664699 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:30.664711 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:30.664766 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:30.665040 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:30.665041 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:30.665042 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 20:19:30.665110 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:30.665112 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.665685 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.665686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:30.666616 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:30.666618 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.666999 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.666999 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:30.667066 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:30.667090 (+ 24us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 20:19:30.667289 (+ 199us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 20:19:30.667291 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 20:19:30.667346 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 20:19:30.667387 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260928 20:19:30.676995 4263 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:19:30.677121 4263 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:19:30.677793 4263 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3b2eb202db314d8e8d097278b55d724d: Sending DeleteTablet for 3 replicas of tablet 38d17bed4b8a466b9be4c0eee1d8f5df
I20260928 20:19:30.678072 4540 tablet_service.cc:1514] Processing DeleteTablet for tablet 38d17bed4b8a466b9be4c0eee1d8f5df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:19:30 UTC) from {username='kudu', principal='kudu/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:52246
I20260928 20:19:30.678190 4680 tablet_service.cc:1514] Processing DeleteTablet for tablet 38d17bed4b8a466b9be4c0eee1d8f5df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:19:30 UTC) from {username='kudu', principal='kudu/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:37608
I20260928 20:19:30.678282 4400 tablet_service.cc:1514] Processing DeleteTablet for tablet 38d17bed4b8a466b9be4c0eee1d8f5df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:19:30 UTC) from {username='kudu', principal='kudu/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:47478
I20260928 20:19:30.678372 4792 tablet_replica.cc:330] T 38d17bed4b8a466b9be4c0eee1d8f5df P 13dd5ff0fc59493a84afa5ed4f71c1b0: stopping tablet replica
I20260928 20:19:30.678426 4792 raft_consensus.cc:2229] T 38d17bed4b8a466b9be4c0eee1d8f5df P 13dd5ff0fc59493a84afa5ed4f71c1b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:19:30.678463 4792 raft_consensus.cc:2258] T 38d17bed4b8a466b9be4c0eee1d8f5df P 13dd5ff0fc59493a84afa5ed4f71c1b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:19:30.678529 4794 tablet_replica.cc:330] T 38d17bed4b8a466b9be4c0eee1d8f5df P 8b1a6b51b8344a45a8f9918d5130860e: stopping tablet replica
I20260928 20:19:30.678573 4794 raft_consensus.cc:2229] T 38d17bed4b8a466b9be4c0eee1d8f5df P 8b1a6b51b8344a45a8f9918d5130860e [term 1 LEADER]: Raft consensus shutting down.
I20260928 20:19:30.678664 4794 raft_consensus.cc:2258] T 38d17bed4b8a466b9be4c0eee1d8f5df P 8b1a6b51b8344a45a8f9918d5130860e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:19:30.678741 4793 tablet_replica.cc:330] T 38d17bed4b8a466b9be4c0eee1d8f5df P b3734cff4fd64fce9d89707d7d6e9e94: stopping tablet replica
I20260928 20:19:30.678813 4793 raft_consensus.cc:2229] T 38d17bed4b8a466b9be4c0eee1d8f5df P b3734cff4fd64fce9d89707d7d6e9e94 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:19:30.678848 4793 raft_consensus.cc:2258] T 38d17bed4b8a466b9be4c0eee1d8f5df P b3734cff4fd64fce9d89707d7d6e9e94 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:19:30.679222 4793 ts_tablet_manager.cc:1905] T 38d17bed4b8a466b9be4c0eee1d8f5df P b3734cff4fd64fce9d89707d7d6e9e94: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:19:30.679489 4792 ts_tablet_manager.cc:1905] T 38d17bed4b8a466b9be4c0eee1d8f5df P 13dd5ff0fc59493a84afa5ed4f71c1b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:19:30.681274 4792 ts_tablet_manager.cc:1918] T 38d17bed4b8a466b9be4c0eee1d8f5df P 13dd5ff0fc59493a84afa5ed4f71c1b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 20:19:30.681348 4792 log.cc:1196] T 38d17bed4b8a466b9be4c0eee1d8f5df P 13dd5ff0fc59493a84afa5ed4f71c1b0: Deleting WAL directory at /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/ts-1/wal/wals/38d17bed4b8a466b9be4c0eee1d8f5df
I20260928 20:19:30.681597 4792 ts_tablet_manager.cc:1939] T 38d17bed4b8a466b9be4c0eee1d8f5df P 13dd5ff0fc59493a84afa5ed4f71c1b0: Deleting consensus metadata
I20260928 20:19:30.681947 4794 ts_tablet_manager.cc:1905] T 38d17bed4b8a466b9be4c0eee1d8f5df P 8b1a6b51b8344a45a8f9918d5130860e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:19:30.681979 4793 ts_tablet_manager.cc:1918] T 38d17bed4b8a466b9be4c0eee1d8f5df P b3734cff4fd64fce9d89707d7d6e9e94: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 20:19:30.682035 4793 log.cc:1196] T 38d17bed4b8a466b9be4c0eee1d8f5df P b3734cff4fd64fce9d89707d7d6e9e94: Deleting WAL directory at /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/ts-0/wal/wals/38d17bed4b8a466b9be4c0eee1d8f5df
I20260928 20:19:30.682267 4793 ts_tablet_manager.cc:1939] T 38d17bed4b8a466b9be4c0eee1d8f5df P b3734cff4fd64fce9d89707d7d6e9e94: Deleting consensus metadata
I20260928 20:19:30.682662 4250 catalog_manager.cc:4841] TS b3734cff4fd64fce9d89707d7d6e9e94 (127.3.18.1:40669): tablet 38d17bed4b8a466b9be4c0eee1d8f5df (table test-table [id=fb5b21539dff412fbd81253697c3b220]) successfully deleted
I20260928 20:19:30.683103 4248 catalog_manager.cc:4841] TS 13dd5ff0fc59493a84afa5ed4f71c1b0 (127.3.18.2:33777): tablet 38d17bed4b8a466b9be4c0eee1d8f5df (table test-table [id=fb5b21539dff412fbd81253697c3b220]) successfully deleted
I20260928 20:19:30.683632 4794 ts_tablet_manager.cc:1918] T 38d17bed4b8a466b9be4c0eee1d8f5df P 8b1a6b51b8344a45a8f9918d5130860e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 20:19:30.683702 4794 log.cc:1196] T 38d17bed4b8a466b9be4c0eee1d8f5df P 8b1a6b51b8344a45a8f9918d5130860e: Deleting WAL directory at /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790626764250411-3144-0/minicluster-data/ts-2/wal/wals/38d17bed4b8a466b9be4c0eee1d8f5df
I20260928 20:19:30.683948 4794 ts_tablet_manager.cc:1939] T 38d17bed4b8a466b9be4c0eee1d8f5df P 8b1a6b51b8344a45a8f9918d5130860e: Deleting consensus metadata
I20260928 20:19:30.684291 4250 catalog_manager.cc:4841] TS 8b1a6b51b8344a45a8f9918d5130860e (127.3.18.3:46789): tablet 38d17bed4b8a466b9be4c0eee1d8f5df (table test-table [id=fb5b21539dff412fbd81253697c3b220]) successfully deleted
Sep 28 20:19:30 dist-test-slave-b83v krb5kdc[4218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626770, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.18.1@KRBTEST.COM
I20260928 20:19:30.700389 4768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:30.683481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47484 (local address 127.3.18.1:40669)
0928 20:19:30.683528 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:30.683530 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:30.683921 (+ 391us) server_negotiation.cc:407] Connection header received
0928 20:19:30.684088 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:30.684090 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:30.684101 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:30.684147 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:30.685015 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.685798 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.686926 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.687248 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.687328 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:30.696469 (+ 9141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:30.696485 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:30.696486 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:30.696497 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:30.698749 (+ 2252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:30.699238 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:30.699243 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:30.699243 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:30.699319 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:30.699539 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:30.699540 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:30.699541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:30.699737 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:30.699983 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:30.700079 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 20:19:30.700178 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 20:19:30.700652 4442 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47484
I20260928 20:19:30.709591 4312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:30.701684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51812 (local address 127.3.18.62:40329)
0928 20:19:30.701735 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:30.701737 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:30.701964 (+ 227us) server_negotiation.cc:407] Connection header received
0928 20:19:30.702030 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:30.702031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:30.702043 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:30.702116 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:30.702764 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.703521 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.704614 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:30.704924 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:30.705014 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:30.706524 (+ 1510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:30.706526 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:30.706526 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:30.706534 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:30.708292 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:30.708724 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:30.708726 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:30.708726 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:30.708793 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:30.709018 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:30.709019 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:30.709019 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:30.709173 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:30.709290 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:30.709404 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 20:19:30.709490 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
W20260928 20:19:30.710057 4263 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51812
I20260928 20:19:30.711071 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 4335
I20260928 20:19:30.720325 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 4475
I20260928 20:19:30.726257 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 4615
I20260928 20:19:30.731719 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 4231
2026-09-28T20:19:30Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2288 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:19:30 dist-test-slave-b83v krb5kdc[4833](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:19:30 dist-test-slave-b83v krb5kdc[4833](info): set up 2 sockets
Sep 28 20:19:30 dist-test-slave-b83v krb5kdc[4833](info): commencing operation
krb5kdc: starting...
W20260928 20:19:31.832054 3144 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Sep 28 20:19:31 dist-test-slave-b83v krb5kdc[4833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:19:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:19:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.18.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab.
Entry for principal kudu/127.3.18.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab.
I20260928 20:19:31.960714 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:39065
--webserver_interface=127.3.18.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:37153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.18.62:39065
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:19:32.043500 4846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:19:32.043819 4846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:19:32.043872 4846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:19:32.045141 4846 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:19:32.045223 4846 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 20:19:32.045245 4846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:19:32.045269 4846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:19:32.045298 4846 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:19:32.045326 4846 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:19:32.046969 4846 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:37153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.18.62:39065
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:39065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.18.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:19:32.047295 4846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:19:32.047554 4846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:19:32.052829 4851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:32.052837 4852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:32.053043 4846 server_base.cc:1034] running on GCE node
W20260928 20:19:32.052836 4854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:32.053521 4846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:19:32.053843 4846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:19:32.054989 4846 hybrid_clock.cc:648] HybridClock initialized: now 1790626772054972 us; error 43 us; skew 500 ppm
Sep 28 20:19:32 dist-test-slave-b83v krb5kdc[4833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626772, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:19:32.057790 4846 init.cc:377] Logged in from keytab as kudu/127.3.18.62@KRBTEST.COM (short username kudu)
I20260928 20:19:32.058334 4846 webserver.cc:460] Webserver started at http://127.3.18.62:35839/ using document root <none> and password file <none>
I20260928 20:19:32.058586 4846 fs_manager.cc:359] Metadata directory not provided
I20260928 20:19:32.058653 4846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:19:32.058804 4846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:19:32.060106 4846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/master-0/data/instance:
uuid: "cddefad8370040909095f66cc0817b0e"
format_stamp: "Formatted at 2026-09-28 20:19:32 on dist-test-slave-b83v"
server_key: "6f5ba1bd6f9cd9dfd305d28f64b75784"
server_key_iv: "68aebd326f6d3faf7c8f17554aef4b1f"
server_key_version: "encryptionkey@0"
I20260928 20:19:32.060491 4846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/master-0/wal/instance:
uuid: "cddefad8370040909095f66cc0817b0e"
format_stamp: "Formatted at 2026-09-28 20:19:32 on dist-test-slave-b83v"
server_key: "6f5ba1bd6f9cd9dfd305d28f64b75784"
server_key_iv: "68aebd326f6d3faf7c8f17554aef4b1f"
server_key_version: "encryptionkey@0"
I20260928 20:19:32.061946 4846 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 20:19:32.062860 4861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:32.063097 4846 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:19:32.063300 4846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/master-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/master-0/wal
uuid: "cddefad8370040909095f66cc0817b0e"
format_stamp: "Formatted at 2026-09-28 20:19:32 on dist-test-slave-b83v"
server_key: "6f5ba1bd6f9cd9dfd305d28f64b75784"
server_key_iv: "68aebd326f6d3faf7c8f17554aef4b1f"
server_key_version: "encryptionkey@0"
I20260928 20:19:32.063383 4846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:19:32.089910 4846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:19:32.092211 4846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:19:32.092381 4846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:19:32.097268 4846 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.62:39065
I20260928 20:19:32.097328 4923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.62:39065 every 8 connection(s)
I20260928 20:19:32.097669 4846 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
I20260928 20:19:32.098309 4924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:32.100729 4924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cddefad8370040909095f66cc0817b0e: Bootstrap starting.
I20260928 20:19:32.101401 4924 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cddefad8370040909095f66cc0817b0e: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:32.101711 4924 log.cc:824] T 00000000000000000000000000000000 P cddefad8370040909095f66cc0817b0e: Log is configured to *not* fsync() on all Append() calls
I20260928 20:19:32.102690 4924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cddefad8370040909095f66cc0817b0e: No bootstrap required, opened a new log
I20260928 20:19:32.104964 4924 raft_consensus.cc:348] T 00000000000000000000000000000000 P cddefad8370040909095f66cc0817b0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cddefad8370040909095f66cc0817b0e" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 39065 } }
I20260928 20:19:32.105118 4924 raft_consensus.cc:374] T 00000000000000000000000000000000 P cddefad8370040909095f66cc0817b0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:32.105141 4924 raft_consensus.cc:729] T 00000000000000000000000000000000 P cddefad8370040909095f66cc0817b0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cddefad8370040909095f66cc0817b0e, State: Initialized, Role: FOLLOWER
I20260928 20:19:32.105298 4924 consensus_queue.cc:260] T 00000000000000000000000000000000 P cddefad8370040909095f66cc0817b0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cddefad8370040909095f66cc0817b0e" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 39065 } }
I20260928 20:19:32.105376 4924 raft_consensus.cc:388] T 00000000000000000000000000000000 P cddefad8370040909095f66cc0817b0e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:19:32.105400 4924 raft_consensus.cc:482] T 00000000000000000000000000000000 P cddefad8370040909095f66cc0817b0e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:19:32.105465 4924 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cddefad8370040909095f66cc0817b0e [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:32.106037 4924 raft_consensus.cc:504] T 00000000000000000000000000000000 P cddefad8370040909095f66cc0817b0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cddefad8370040909095f66cc0817b0e" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 39065 } }
I20260928 20:19:32.106158 4924 leader_election.cc:302] T 00000000000000000000000000000000 P cddefad8370040909095f66cc0817b0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cddefad8370040909095f66cc0817b0e; no voters:
I20260928 20:19:32.106348 4924 leader_election.cc:288] T 00000000000000000000000000000000 P cddefad8370040909095f66cc0817b0e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:19:32.106423 4927 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cddefad8370040909095f66cc0817b0e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:19:32.106575 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 4846
I20260928 20:19:32.106632 4924 sys_catalog.cc:564] T 00000000000000000000000000000000 P cddefad8370040909095f66cc0817b0e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:19:32.106679 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/master-0/wal/instance
I20260928 20:19:32.106707 4927 raft_consensus.cc:686] T 00000000000000000000000000000000 P cddefad8370040909095f66cc0817b0e [term 1 LEADER]: Becoming Leader. State: Replica: cddefad8370040909095f66cc0817b0e, State: Running, Role: LEADER
I20260928 20:19:32.106827 4927 consensus_queue.cc:237] T 00000000000000000000000000000000 P cddefad8370040909095f66cc0817b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cddefad8370040909095f66cc0817b0e" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 39065 } }
I20260928 20:19:32.106894 3144 external_mini_cluster.cc:1428] Setting key 45718b9745b6f3f5f92ff8a54e9d7dae
I20260928 20:19:32.107429 4929 sys_catalog.cc:455] T 00000000000000000000000000000000 P cddefad8370040909095f66cc0817b0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cddefad8370040909095f66cc0817b0e. Latest consensus state: current_term: 1 leader_uuid: "cddefad8370040909095f66cc0817b0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cddefad8370040909095f66cc0817b0e" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 39065 } } }
I20260928 20:19:32.107426 4928 sys_catalog.cc:455] T 00000000000000000000000000000000 P cddefad8370040909095f66cc0817b0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cddefad8370040909095f66cc0817b0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cddefad8370040909095f66cc0817b0e" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 39065 } } }
I20260928 20:19:32.107537 4929 sys_catalog.cc:458] T 00000000000000000000000000000000 P cddefad8370040909095f66cc0817b0e [sys.catalog]: This master's current role is: LEADER
I20260928 20:19:32.107546 4928 sys_catalog.cc:458] T 00000000000000000000000000000000 P cddefad8370040909095f66cc0817b0e [sys.catalog]: This master's current role is: LEADER
I20260928 20:19:32.107883 4933 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:19:32.108435 4933 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:19:32.110195 4933 catalog_manager.cc:1347] Generated new cluster ID: 0dbf179e5f44426d8ed8ef98807c03b8
I20260928 20:19:32.110262 4933 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 20:19:32 dist-test-slave-b83v krb5kdc[4833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:19:32.118502 4932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:32.107537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49064 (local address 127.3.18.62:39065)
0928 20:19:32.107734 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:32.107747 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:32.107807 (+ 60us) server_negotiation.cc:407] Connection header received
0928 20:19:32.107946 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:32.107955 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:32.108028 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:32.109181 (+ 1153us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:32.109261 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:32.110497 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:32.111563 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:32.112082 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:32.112184 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:32.114806 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:32.114810 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:32.114811 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:32.114823 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:32.116813 (+ 1990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:32.117226 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:32.117229 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:32.117229 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:32.117354 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:32.117580 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:32.117580 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:32.117581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:32.117872 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:32.118009 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:32.118153 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 20:19:32.118288 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":79,"threads_started":1}
I20260928 20:19:32.119859 4933 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:19:32.120388 4933 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:19:32.126493 4933 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cddefad8370040909095f66cc0817b0e: Generated new TSK 0
I20260928 20:19:32.126696 4933 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.18.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab.
Entry for principal kudu/127.3.18.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab.
I20260928 20:19:32.159996 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.1:0
--local_ip_for_outbound_sockets=127.3.18.1
--webserver_interface=127.3.18.1
--webserver_port=0
--tserver_master_addrs=127.3.18.62:39065
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab
--principal=kudu/127.3.18.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:37153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:19:32.253669 4950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:19:32.253886 4950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:19:32.253916 4950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:19:32.255527 4950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:19:32.255584 4950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:19:32.255606 4950 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 20:19:32.255621 4950 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 20:19:32.255640 4950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.1
I20260928 20:19:32.257319 4950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:37153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab
--principal=kudu/127.3.18.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.18.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.18.62:39065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.1
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:19:32.257529 4950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:19:32.257725 4950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:19:32.263113 4956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:32.263293 4950 server_base.cc:1034] running on GCE node
W20260928 20:19:32.263231 4955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:32.263135 4958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:32.263648 4950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:19:32.263972 4950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:19:32.265180 4950 hybrid_clock.cc:648] HybridClock initialized: now 1790626772265149 us; error 71 us; skew 500 ppm
Sep 28 20:19:32 dist-test-slave-b83v krb5kdc[4833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626772, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:19:32.268273 4950 init.cc:377] Logged in from keytab as kudu/127.3.18.1@KRBTEST.COM (short username kudu)
I20260928 20:19:32.268697 4950 webserver.cc:460] Webserver started at http://127.3.18.1:39957/ using document root <none> and password file <none>
I20260928 20:19:32.268942 4950 fs_manager.cc:359] Metadata directory not provided
I20260928 20:19:32.268989 4950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:19:32.269156 4950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:19:32.270244 4950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/ts-0/data/instance:
uuid: "2b229b3bf9b84796bb23853b0600c783"
format_stamp: "Formatted at 2026-09-28 20:19:32 on dist-test-slave-b83v"
server_key: "94106d9567a42e7287ee816b2bbf171f"
server_key_iv: "5709643f0e4408a060a0d5944cb80f66"
server_key_version: "encryptionkey@0"
I20260928 20:19:32.270601 4950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/ts-0/wal/instance:
uuid: "2b229b3bf9b84796bb23853b0600c783"
format_stamp: "Formatted at 2026-09-28 20:19:32 on dist-test-slave-b83v"
server_key: "94106d9567a42e7287ee816b2bbf171f"
server_key_iv: "5709643f0e4408a060a0d5944cb80f66"
server_key_version: "encryptionkey@0"
I20260928 20:19:32.272044 4950 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:19:32.272861 4965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:32.273032 4950 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 20:19:32.273134 4950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/ts-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/ts-0/wal
uuid: "2b229b3bf9b84796bb23853b0600c783"
format_stamp: "Formatted at 2026-09-28 20:19:32 on dist-test-slave-b83v"
server_key: "94106d9567a42e7287ee816b2bbf171f"
server_key_iv: "5709643f0e4408a060a0d5944cb80f66"
server_key_version: "encryptionkey@0"
I20260928 20:19:32.273264 4950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:19:32.283478 4950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:19:32.285950 4950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:19:32.286116 4950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:19:32.286621 4950 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:19:32.286684 4950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:32.286757 4950 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:19:32.286795 4950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:32.293615 4950 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.1:42595
I20260928 20:19:32.293696 5080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.1:42595 every 8 connection(s)
I20260928 20:19:32.294044 4950 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
Sep 28 20:19:32 dist-test-slave-b83v krb5kdc[4833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626772, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.1@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:19:32.295403 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 4950
I20260928 20:19:32.295512 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/ts-0/wal/instance
I20260928 20:19:32.295696 3144 external_mini_cluster.cc:1428] Setting key be3a47bf4d8e0458adc4ab4101953d35
I20260928 20:19:32.298739 4932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:32.287250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:60287 (local address 127.3.18.62:39065)
0928 20:19:32.287314 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:32.287316 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:32.287499 (+ 183us) server_negotiation.cc:407] Connection header received
0928 20:19:32.288174 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:32.288176 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:32.288191 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:32.288273 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:32.289149 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:32.290021 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:32.291732 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:32.292129 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:32.292238 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:32.295001 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:32.295003 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:32.295003 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:32.295011 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:32.297268 (+ 2257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:32.297701 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:32.297703 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:32.297703 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:32.297771 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:32.298036 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:32.298037 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:32.298037 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:32.298184 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:32.298261 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:32.298547 (+ 286us) server_negotiation.cc:299] Negotiation successful
0928 20:19:32.298647 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 20:19:32.298941 4974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:32.287176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:39065 (local address 127.3.18.1:60287)
0928 20:19:32.287368 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:32.287414 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:32.288036 (+ 622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:32.288355 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:32.288359 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:32.288728 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:32.289039 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:32.289044 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:32.290146 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:32.290147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:32.291615 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:32.291618 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:32.292270 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:32.292271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:32.292434 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:32.293101 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:19:32.293111 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:32.294892 (+ 1781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:32.297357 (+ 2465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:32.297363 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:32.297365 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:32.297610 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:32.297847 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:32.297847 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:32.297848 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:32.297963 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:32.298262 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:32.298270 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:32.298414 (+ 144us) client_negotiation.cc:769] Sending connection context
0928 20:19:32.298522 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 20:19:32.298689 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":90,"threads_started":1}
I20260928 20:19:32.299467 5081 heartbeater.cc:347] Connected to a master server at 127.3.18.62:39065
I20260928 20:19:32.299615 5081 heartbeater.cc:464] Registering TS with master...
I20260928 20:19:32.299849 5081 heartbeater.cc:511] Master 127.3.18.62:39065 requested a full tablet report, sending...
I20260928 20:19:32.300427 4877 ts_manager.cc:194] Registered new tserver with Master: 2b229b3bf9b84796bb23853b0600c783 (127.3.18.1:42595)
I20260928 20:19:32.301537 4877 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.18.1@KRBTEST.COM'} at 127.3.18.1:60287
WARNING: no policy specified for kudu/127.3.18.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.2@KRBTEST.COM" created.
I20260928 20:19:32.310899 4932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:32.300796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:56735 (local address 127.3.18.62:39065)
0928 20:19:32.300836 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:32.300838 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:32.304017 (+ 3179us) server_negotiation.cc:407] Connection header received
0928 20:19:32.304152 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:32.304153 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:32.304200 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:32.304303 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:32.304677 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:32.305194 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:32.305964 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:32.306226 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:32.306295 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:32.307327 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:32.307329 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:32.307329 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:32.307337 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:32.309481 (+ 2144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:32.309908 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:32.309909 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:32.309910 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:32.309982 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:32.310191 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:32.310192 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:32.310192 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:32.310360 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:32.310436 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:32.310637 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 20:19:32.310738 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 20:19:32.311429 5088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:32.300625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:39065 (local address 127.3.18.1:56735)
0928 20:19:32.303940 (+ 3315us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:32.303951 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:32.304033 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:32.304311 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:32.304313 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:32.304528 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:32.304586 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:32.304589 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:32.305291 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:32.305292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:32.305835 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:32.305837 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:32.306293 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:32.306294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:32.306375 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:32.306841 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:19:32.306846 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:32.307234 (+ 388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:32.309582 (+ 2348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:32.309583 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:32.309584 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:32.309820 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:32.310060 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:32.310060 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:32.310060 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:32.310119 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:32.310464 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:32.310465 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:32.310520 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 20:19:32.310589 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 20:19:32.310715 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3286,"thread_start_us":86,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab.
Entry for principal kudu/127.3.18.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab.
I20260928 20:19:32.328691 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.2:0
--local_ip_for_outbound_sockets=127.3.18.2
--webserver_interface=127.3.18.2
--webserver_port=0
--tserver_master_addrs=127.3.18.62:39065
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab
--principal=kudu/127.3.18.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:37153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:19:32.417079 5090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:19:32.417308 5090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:19:32.417335 5090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:19:32.418509 5090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:19:32.418553 5090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:19:32.418574 5090 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 20:19:32.418586 5090 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 20:19:32.418603 5090 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.2
I20260928 20:19:32.420120 5090 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:37153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab
--principal=kudu/127.3.18.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.18.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.18.62:39065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.2
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:19:32.420389 5090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:19:32.420648 5090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:19:32.426041 5095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:32.426091 5096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:32.426041 5098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:32.426337 5090 server_base.cc:1034] running on GCE node
I20260928 20:19:32.426568 5090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:19:32.426877 5090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:19:32.428040 5090 hybrid_clock.cc:648] HybridClock initialized: now 1790626772428027 us; error 36 us; skew 500 ppm
Sep 28 20:19:32 dist-test-slave-b83v krb5kdc[4833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626772, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:19:32.430759 5090 init.cc:377] Logged in from keytab as kudu/127.3.18.2@KRBTEST.COM (short username kudu)
I20260928 20:19:32.431223 5090 webserver.cc:460] Webserver started at http://127.3.18.2:38227/ using document root <none> and password file <none>
I20260928 20:19:32.431468 5090 fs_manager.cc:359] Metadata directory not provided
I20260928 20:19:32.431599 5090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:19:32.431831 5090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:19:32.432934 5090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/ts-1/data/instance:
uuid: "ba5667d185cb4f0e8d923ae42f7105bf"
format_stamp: "Formatted at 2026-09-28 20:19:32 on dist-test-slave-b83v"
server_key: "cc3fefb17e06847e22496aa588b9fec4"
server_key_iv: "51dc6739a3a35fb5f4bc40eae75ee5ed"
server_key_version: "encryptionkey@0"
I20260928 20:19:32.433619 5090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/ts-1/wal/instance:
uuid: "ba5667d185cb4f0e8d923ae42f7105bf"
format_stamp: "Formatted at 2026-09-28 20:19:32 on dist-test-slave-b83v"
server_key: "cc3fefb17e06847e22496aa588b9fec4"
server_key_iv: "51dc6739a3a35fb5f4bc40eae75ee5ed"
server_key_version: "encryptionkey@0"
I20260928 20:19:32.435030 5090 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:19:32.435988 5105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:32.436287 5090 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:19:32.436399 5090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/ts-1/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/ts-1/wal
uuid: "ba5667d185cb4f0e8d923ae42f7105bf"
format_stamp: "Formatted at 2026-09-28 20:19:32 on dist-test-slave-b83v"
server_key: "cc3fefb17e06847e22496aa588b9fec4"
server_key_iv: "51dc6739a3a35fb5f4bc40eae75ee5ed"
server_key_version: "encryptionkey@0"
I20260928 20:19:32.436477 5090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:19:32.449230 5090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:19:32.451475 5090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:19:32.451633 5090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:19:32.452231 5090 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:19:32.452278 5090 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:32.452302 5090 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:19:32.452338 5090 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:32.459340 5090 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.2:34589
I20260928 20:19:32.459379 5220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.2:34589 every 8 connection(s)
I20260928 20:19:32.459810 5090 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
Sep 28 20:19:32 dist-test-slave-b83v krb5kdc[4833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626772, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.2@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:19:32.463876 4932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:32.452864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:57777 (local address 127.3.18.62:39065)
0928 20:19:32.452928 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:32.452931 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:32.453214 (+ 283us) server_negotiation.cc:407] Connection header received
0928 20:19:32.453863 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:32.453864 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:32.453878 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:32.453949 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:32.454887 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:32.455792 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:32.457381 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:32.457756 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:32.457860 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:32.460928 (+ 3068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:32.460929 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:32.460930 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:32.460938 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:32.462395 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:32.462865 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:32.462866 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:32.462867 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:32.462929 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:32.463193 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:32.463194 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:32.463194 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:32.463316 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:32.463385 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:32.463667 (+ 282us) server_negotiation.cc:299] Negotiation successful
0928 20:19:32.463754 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 20:19:32.464104 5114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:32.452882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:39065 (local address 127.3.18.2:57777)
0928 20:19:32.453078 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:32.453116 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:32.453720 (+ 604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:32.454075 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:32.454080 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:32.454481 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:32.454773 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:32.454778 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:32.455895 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:32.455896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:32.457294 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:32.457296 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:32.457873 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:32.457874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:32.458065 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:32.458776 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:19:32.458786 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:32.460808 (+ 2022us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:32.462518 (+ 1710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:32.462525 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:32.462528 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:32.462773 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:32.462994 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:32.462995 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:32.462995 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:32.463114 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:32.463385 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:32.463393 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:32.463535 (+ 142us) client_negotiation.cc:769] Sending connection context
0928 20:19:32.463643 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 20:19:32.463845 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":92,"threads_started":1}
I20260928 20:19:32.464175 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 5090
I20260928 20:19:32.464265 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/ts-1/wal/instance
I20260928 20:19:32.464443 3144 external_mini_cluster.cc:1428] Setting key e615c59b542cae540863408fa293d4ee
I20260928 20:19:32.464988 5221 heartbeater.cc:347] Connected to a master server at 127.3.18.62:39065
I20260928 20:19:32.465119 5221 heartbeater.cc:464] Registering TS with master...
I20260928 20:19:32.465382 5221 heartbeater.cc:511] Master 127.3.18.62:39065 requested a full tablet report, sending...
I20260928 20:19:32.465860 4878 ts_manager.cc:194] Registered new tserver with Master: ba5667d185cb4f0e8d923ae42f7105bf (127.3.18.2:34589)
I20260928 20:19:32.466312 4878 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.18.2@KRBTEST.COM'} at 127.3.18.2:57777
I20260928 20:19:32.472499 4932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:32.465849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:56441 (local address 127.3.18.62:39065)
0928 20:19:32.465893 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:32.465895 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:32.466761 (+ 866us) server_negotiation.cc:407] Connection header received
0928 20:19:32.466815 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:32.466816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:32.466826 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:32.466887 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:32.467339 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:32.467920 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:32.468696 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:32.468899 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:32.468984 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:32.469970 (+ 986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:32.469972 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:32.469972 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:32.469979 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:32.471340 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:32.471686 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:32.471687 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:32.471688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:32.471748 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:32.472002 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:32.472003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:32.472003 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:32.472140 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:32.472201 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:32.472374 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 20:19:32.472434 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 20:19:32.472590 5228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:32.466512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:39065 (local address 127.3.18.2:56441)
0928 20:19:32.466670 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:32.466684 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:32.466764 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:32.466902 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:32.466903 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:32.467134 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:32.467221 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:32.467223 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:32.468015 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:32.468016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:32.468602 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:32.468604 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:32.469012 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:32.469013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:32.469082 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:32.469559 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:19:32.469564 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:32.469866 (+ 302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:32.471438 (+ 1572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:32.471440 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:32.471440 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:32.471612 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:32.471853 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:32.471855 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:32.471855 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:32.471932 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:32.472213 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:32.472214 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:32.472267 (+ 53us) client_negotiation.cc:769] Sending connection context
0928 20:19:32.472355 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 20:19:32.472469 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":669,"threads_started":1}
WARNING: no policy specified for kudu/127.3.18.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab.
Entry for principal kudu/127.3.18.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab.
I20260928 20:19:32.495234 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.3:0
--local_ip_for_outbound_sockets=127.3.18.3
--webserver_interface=127.3.18.3
--webserver_port=0
--tserver_master_addrs=127.3.18.62:39065
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab
--principal=kudu/127.3.18.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:37153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:19:32.580166 5230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:19:32.580386 5230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:19:32.580413 5230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:19:32.581624 5230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:19:32.581672 5230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:19:32.581694 5230 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 20:19:32.581707 5230 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 20:19:32.581723 5230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.3
I20260928 20:19:32.583267 5230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:37153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab
--principal=kudu/127.3.18.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.18.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.18.62:39065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.3
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:19:32.583534 5230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:19:32.583807 5230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:19:32.589219 5235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:32.589236 5236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:32.589421 5230 server_base.cc:1034] running on GCE node
W20260928 20:19:32.589468 5238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:32.589725 5230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:19:32.590039 5230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:19:32.591209 5230 hybrid_clock.cc:648] HybridClock initialized: now 1790626772591188 us; error 38 us; skew 500 ppm
Sep 28 20:19:32 dist-test-slave-b83v krb5kdc[4833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626772, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:19:32.594012 5230 init.cc:377] Logged in from keytab as kudu/127.3.18.3@KRBTEST.COM (short username kudu)
I20260928 20:19:32.594548 5230 webserver.cc:460] Webserver started at http://127.3.18.3:34443/ using document root <none> and password file <none>
I20260928 20:19:32.594790 5230 fs_manager.cc:359] Metadata directory not provided
I20260928 20:19:32.594857 5230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:19:32.595007 5230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:19:32.596228 5230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/ts-2/data/instance:
uuid: "ae62d02eff7749979f0c98a8907c918a"
format_stamp: "Formatted at 2026-09-28 20:19:32 on dist-test-slave-b83v"
server_key: "db92a966c4ea6be0229428a28729d599"
server_key_iv: "d77bfd167e26f8f8fbd7dfdce6aea126"
server_key_version: "encryptionkey@0"
I20260928 20:19:32.596594 5230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/ts-2/wal/instance:
uuid: "ae62d02eff7749979f0c98a8907c918a"
format_stamp: "Formatted at 2026-09-28 20:19:32 on dist-test-slave-b83v"
server_key: "db92a966c4ea6be0229428a28729d599"
server_key_iv: "d77bfd167e26f8f8fbd7dfdce6aea126"
server_key_version: "encryptionkey@0"
I20260928 20:19:32.598026 5230 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 20:19:32.598935 5245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:32.599153 5230 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:19:32.599263 5230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/ts-2/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/ts-2/wal
uuid: "ae62d02eff7749979f0c98a8907c918a"
format_stamp: "Formatted at 2026-09-28 20:19:32 on dist-test-slave-b83v"
server_key: "db92a966c4ea6be0229428a28729d599"
server_key_iv: "d77bfd167e26f8f8fbd7dfdce6aea126"
server_key_version: "encryptionkey@0"
I20260928 20:19:32.599346 5230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:19:32.620306 5230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:19:32.622851 5230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:19:32.623037 5230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:19:32.623695 5230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:19:32.623749 5230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:32.623835 5230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:19:32.623871 5230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 20:19:32 dist-test-slave-b83v krb5kdc[4833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626772, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.3@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:19:32.631314 5230 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.3:38807
I20260928 20:19:32.631466 5360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.3:38807 every 8 connection(s)
I20260928 20:19:32.631737 5230 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
I20260928 20:19:32.634578 4932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:32.624194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:37301 (local address 127.3.18.62:39065)
0928 20:19:32.624261 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:32.624263 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:32.624502 (+ 239us) server_negotiation.cc:407] Connection header received
0928 20:19:32.625014 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:32.625015 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:32.625031 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:32.625145 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:32.625821 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:32.626407 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:32.628245 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:32.628556 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:32.628654 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:32.631538 (+ 2884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:32.631540 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:32.631540 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:32.631549 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:32.633156 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:32.633597 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:32.633598 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:32.633599 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:32.633661 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:32.633965 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:32.633965 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:32.633966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:32.634079 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:32.634131 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:32.634444 (+ 313us) server_negotiation.cc:299] Negotiation successful
0928 20:19:32.634513 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 20:19:32.634806 5254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:32.624183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:39065 (local address 127.3.18.3:37301)
0928 20:19:32.624363 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:32.624405 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:32.624913 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:32.625241 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:32.625246 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:32.625509 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:32.625702 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:32.625707 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:32.626517 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:32.626518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:32.628085 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:32.628088 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:32.628738 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:32.628739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:32.628851 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:32.629409 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:19:32.629420 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:32.631419 (+ 1999us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:32.633241 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:32.633247 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:32.633250 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:32.633475 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:32.633782 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:32.633784 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:32.633784 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:32.633898 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:32.634192 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:32.634196 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:32.634320 (+ 124us) client_negotiation.cc:769] Sending connection context
0928 20:19:32.634412 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 20:19:32.634546 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":107,"threads_started":1}
I20260928 20:19:32.635304 5361 heartbeater.cc:347] Connected to a master server at 127.3.18.62:39065
I20260928 20:19:32.635389 5361 heartbeater.cc:464] Registering TS with master...
I20260928 20:19:32.635582 5361 heartbeater.cc:511] Master 127.3.18.62:39065 requested a full tablet report, sending...
I20260928 20:19:32.636019 4877 ts_manager.cc:194] Registered new tserver with Master: ae62d02eff7749979f0c98a8907c918a (127.3.18.3:38807)
I20260928 20:19:32.636543 4877 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.18.3@KRBTEST.COM'} at 127.3.18.3:37301
I20260928 20:19:32.640733 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 5230
I20260928 20:19:32.640843 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/ts-2/wal/instance
I20260928 20:19:32.641044 3144 external_mini_cluster.cc:1428] Setting key f1b8834ceec041ca08be0288ad03ffb3
I20260928 20:19:32.641987 3144 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 20:19:32.642587 4932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:32.636425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:32859 (local address 127.3.18.62:39065)
0928 20:19:32.636469 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:32.636470 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:32.636599 (+ 129us) server_negotiation.cc:407] Connection header received
0928 20:19:32.636678 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:32.636679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:32.636689 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:32.636737 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:32.637099 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:32.637580 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:32.638332 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:32.638548 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:32.638637 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:32.639714 (+ 1077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:32.639715 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:32.639715 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:32.639723 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:32.641387 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:32.641804 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:32.641805 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:32.641805 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:32.641864 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:32.642060 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:32.642061 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:32.642061 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:32.642180 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:32.642252 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:32.642410 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 20:19:32.642494 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 20:19:32.642602 5367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:32.636363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:39065 (local address 127.3.18.3:32859)
0928 20:19:32.636530 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:32.636539 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:32.636598 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:32.636753 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:32.636754 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:32.636955 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:32.637008 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:32.637010 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:32.637673 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:32.637674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:32.638231 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:32.638233 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:32.638637 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:32.638639 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:32.638716 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:32.639266 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:19:32.639271 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:32.639609 (+ 338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:32.641478 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:32.641479 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:32.641479 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:32.641732 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:32.641933 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:32.641934 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:32.641934 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:32.641994 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:32.642244 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:32.642244 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:32.642291 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 20:19:32.642385 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 20:19:32.642495 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":75,"threads_started":1}
Sep 28 20:19:32 dist-test-slave-b83v krb5kdc[4833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 20:19:32 dist-test-slave-b83v krb5kdc[4833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:19:32.666383 4932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:32.658334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60422 (local address 127.3.18.62:39065)
0928 20:19:32.658503 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:32.658505 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:32.658515 (+ 10us) server_negotiation.cc:407] Connection header received
0928 20:19:32.658543 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:32.658553 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:32.658566 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:32.658622 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:32.659286 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:32.659837 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:32.660674 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:32.660865 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:32.660934 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:32.663357 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:32.663359 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:32.663359 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:32.663366 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:32.664983 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:32.665404 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:32.665405 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:32.665406 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:32.665463 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:32.665718 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:32.665719 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:32.665720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:32.665893 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:32.665987 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:32.666219 (+ 232us) server_negotiation.cc:299] Negotiation successful
0928 20:19:32.666291 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260928 20:19:32.668327 4877 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60422:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 20:19:32.670066 4877 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 20:19:32.677271 5380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:32.672649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.3:38807 (local address 127.0.0.1:48108)
0928 20:19:32.673296 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:32.673305 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:32.673357 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:32.673762 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:32.673763 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:32.673765 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:19:32.673804 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:32.673805 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:32.674647 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:32.674648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:32.675933 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:32.675936 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:32.676957 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:32.676958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:32.677039 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:32.677054 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 20:19:32.677120 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 20:19:32.677180 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"mutex_wait_us":2,"thread_start_us":87,"threads_started":1}
I20260928 20:19:32.677325 5382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:32.672890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49984 (local address 127.3.18.2:34589)
0928 20:19:32.673064 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:32.673068 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:32.673135 (+ 67us) server_negotiation.cc:407] Connection header received
0928 20:19:32.673303 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:32.673306 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:32.673327 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:32.673412 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:19:32.673598 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:32.674789 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:32.676064 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:32.676817 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:32.676866 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:32.676923 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:32.677161 (+ 238us) server_negotiation.cc:299] Negotiation successful
0928 20:19:32.677202 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":83,"threads_started":1}
I20260928 20:19:32.677536 5379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:32.672649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.2:34589 (local address 127.0.0.1:49984)
0928 20:19:32.673052 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:32.673068 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:32.673143 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:32.673451 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:32.673455 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:32.673458 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:19:32.673536 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:32.673538 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:32.674899 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:32.674900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:32.675953 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:32.675955 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:32.676986 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:32.676987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:32.677059 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:32.677064 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 20:19:32.677417 (+ 353us) client_negotiation.cc:240] Negotiation successful
0928 20:19:32.677469 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"mutex_wait_us":31,"thread_start_us":91,"threads_started":1}
I20260928 20:19:32.677647 5383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:32.672880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48108 (local address 127.3.18.3:38807)
0928 20:19:32.673137 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:32.673140 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:32.673503 (+ 363us) server_negotiation.cc:407] Connection header received
0928 20:19:32.673651 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:32.673656 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:32.673677 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:32.673758 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:19:32.673860 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:32.674542 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:32.676046 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:32.676789 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:32.676861 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:32.676921 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:32.677499 (+ 578us) server_negotiation.cc:299] Negotiation successful
0928 20:19:32.677542 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":97,"threads_started":1}
I20260928 20:19:32.677973 5378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:32.672622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.1:42595 (local address 127.0.0.1:42410)
0928 20:19:32.672805 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:32.672850 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:32.672914 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:32.673700 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:32.673700 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:32.673702 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:19:32.673750 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:32.673752 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:32.674966 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:32.674967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:32.676161 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:32.676163 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:32.677202 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:32.677203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:32.677263 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:32.677266 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 20:19:32.677314 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 20:19:32.677917 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":74,"threads_started":1}
I20260928 20:19:32.678339 5381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:32.672721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42410 (local address 127.3.18.1:42595)
0928 20:19:32.673480 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:32.673485 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:32.673496 (+ 11us) server_negotiation.cc:407] Connection header received
0928 20:19:32.673535 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:32.673538 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:32.673553 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:32.673628 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:19:32.673828 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:32.674888 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:32.676246 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:32.677043 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:32.677107 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:32.677170 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:32.678196 (+ 1026us) server_negotiation.cc:299] Negotiation successful
0928 20:19:32.678239 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":720,"thread_start_us":105,"threads_started":1}
I20260928 20:19:32.678462 5295 tablet_service.cc:1467] Processing CreateTablet for tablet 6869be9d2b8a48b8940932ce2158a165 (DEFAULT_TABLE table=test-table [id=1209d70dcf9b479091536365d16bcfed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:19:32.678880 5295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6869be9d2b8a48b8940932ce2158a165. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:32.679983 5155 tablet_service.cc:1467] Processing CreateTablet for tablet 6869be9d2b8a48b8940932ce2158a165 (DEFAULT_TABLE table=test-table [id=1209d70dcf9b479091536365d16bcfed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:19:32.680344 5155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6869be9d2b8a48b8940932ce2158a165. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:32.680446 5015 tablet_service.cc:1467] Processing CreateTablet for tablet 6869be9d2b8a48b8940932ce2158a165 (DEFAULT_TABLE table=test-table [id=1209d70dcf9b479091536365d16bcfed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:19:32.680696 5015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6869be9d2b8a48b8940932ce2158a165. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:32.682428 5384 tablet_bootstrap.cc:492] T 6869be9d2b8a48b8940932ce2158a165 P ae62d02eff7749979f0c98a8907c918a: Bootstrap starting.
I20260928 20:19:32.683638 5384 tablet_bootstrap.cc:654] T 6869be9d2b8a48b8940932ce2158a165 P ae62d02eff7749979f0c98a8907c918a: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:32.683707 5385 tablet_bootstrap.cc:492] T 6869be9d2b8a48b8940932ce2158a165 P 2b229b3bf9b84796bb23853b0600c783: Bootstrap starting.
I20260928 20:19:32.683995 5384 log.cc:824] T 6869be9d2b8a48b8940932ce2158a165 P ae62d02eff7749979f0c98a8907c918a: Log is configured to *not* fsync() on all Append() calls
I20260928 20:19:32.684219 5386 tablet_bootstrap.cc:492] T 6869be9d2b8a48b8940932ce2158a165 P ba5667d185cb4f0e8d923ae42f7105bf: Bootstrap starting.
I20260928 20:19:32.684391 5385 tablet_bootstrap.cc:654] T 6869be9d2b8a48b8940932ce2158a165 P 2b229b3bf9b84796bb23853b0600c783: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:32.684697 5384 tablet_bootstrap.cc:492] T 6869be9d2b8a48b8940932ce2158a165 P ae62d02eff7749979f0c98a8907c918a: No bootstrap required, opened a new log
I20260928 20:19:32.684757 5385 log.cc:824] T 6869be9d2b8a48b8940932ce2158a165 P 2b229b3bf9b84796bb23853b0600c783: Log is configured to *not* fsync() on all Append() calls
I20260928 20:19:32.684780 5384 ts_tablet_manager.cc:1397] T 6869be9d2b8a48b8940932ce2158a165 P ae62d02eff7749979f0c98a8907c918a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260928 20:19:32.684895 5386 tablet_bootstrap.cc:654] T 6869be9d2b8a48b8940932ce2158a165 P ba5667d185cb4f0e8d923ae42f7105bf: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:32.685207 5386 log.cc:824] T 6869be9d2b8a48b8940932ce2158a165 P ba5667d185cb4f0e8d923ae42f7105bf: Log is configured to *not* fsync() on all Append() calls
I20260928 20:19:32.685894 5386 tablet_bootstrap.cc:492] T 6869be9d2b8a48b8940932ce2158a165 P ba5667d185cb4f0e8d923ae42f7105bf: No bootstrap required, opened a new log
I20260928 20:19:32.685945 5386 ts_tablet_manager.cc:1397] T 6869be9d2b8a48b8940932ce2158a165 P ba5667d185cb4f0e8d923ae42f7105bf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 20:19:32.686527 5384 raft_consensus.cc:348] T 6869be9d2b8a48b8940932ce2158a165 P ae62d02eff7749979f0c98a8907c918a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b229b3bf9b84796bb23853b0600c783" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42595 } } peers { permanent_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34589 } } peers { permanent_uuid: "ae62d02eff7749979f0c98a8907c918a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38807 } }
I20260928 20:19:32.686699 5384 raft_consensus.cc:374] T 6869be9d2b8a48b8940932ce2158a165 P ae62d02eff7749979f0c98a8907c918a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:32.686755 5384 raft_consensus.cc:729] T 6869be9d2b8a48b8940932ce2158a165 P ae62d02eff7749979f0c98a8907c918a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae62d02eff7749979f0c98a8907c918a, State: Initialized, Role: FOLLOWER
I20260928 20:19:32.686873 5384 consensus_queue.cc:260] T 6869be9d2b8a48b8940932ce2158a165 P ae62d02eff7749979f0c98a8907c918a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b229b3bf9b84796bb23853b0600c783" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42595 } } peers { permanent_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34589 } } peers { permanent_uuid: "ae62d02eff7749979f0c98a8907c918a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38807 } }
I20260928 20:19:32.687155 5384 ts_tablet_manager.cc:1428] T 6869be9d2b8a48b8940932ce2158a165 P ae62d02eff7749979f0c98a8907c918a: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 20:19:32.687186 5361 heartbeater.cc:503] Master 127.3.18.62:39065 was elected leader, sending a full tablet report...
I20260928 20:19:32.687857 5385 tablet_bootstrap.cc:492] T 6869be9d2b8a48b8940932ce2158a165 P 2b229b3bf9b84796bb23853b0600c783: No bootstrap required, opened a new log
I20260928 20:19:32.687928 5385 ts_tablet_manager.cc:1397] T 6869be9d2b8a48b8940932ce2158a165 P 2b229b3bf9b84796bb23853b0600c783: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 20:19:32.688062 5386 raft_consensus.cc:348] T 6869be9d2b8a48b8940932ce2158a165 P ba5667d185cb4f0e8d923ae42f7105bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b229b3bf9b84796bb23853b0600c783" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42595 } } peers { permanent_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34589 } } peers { permanent_uuid: "ae62d02eff7749979f0c98a8907c918a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38807 } }
I20260928 20:19:32.688246 5386 raft_consensus.cc:374] T 6869be9d2b8a48b8940932ce2158a165 P ba5667d185cb4f0e8d923ae42f7105bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:32.688293 5386 raft_consensus.cc:729] T 6869be9d2b8a48b8940932ce2158a165 P ba5667d185cb4f0e8d923ae42f7105bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba5667d185cb4f0e8d923ae42f7105bf, State: Initialized, Role: FOLLOWER
I20260928 20:19:32.688437 5386 consensus_queue.cc:260] T 6869be9d2b8a48b8940932ce2158a165 P ba5667d185cb4f0e8d923ae42f7105bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b229b3bf9b84796bb23853b0600c783" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42595 } } peers { permanent_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34589 } } peers { permanent_uuid: "ae62d02eff7749979f0c98a8907c918a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38807 } }
I20260928 20:19:32.688707 5386 ts_tablet_manager.cc:1428] T 6869be9d2b8a48b8940932ce2158a165 P ba5667d185cb4f0e8d923ae42f7105bf: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 20:19:32.688742 5221 heartbeater.cc:503] Master 127.3.18.62:39065 was elected leader, sending a full tablet report...
I20260928 20:19:32.689924 5385 raft_consensus.cc:348] T 6869be9d2b8a48b8940932ce2158a165 P 2b229b3bf9b84796bb23853b0600c783 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b229b3bf9b84796bb23853b0600c783" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42595 } } peers { permanent_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34589 } } peers { permanent_uuid: "ae62d02eff7749979f0c98a8907c918a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38807 } }
I20260928 20:19:32.690060 5385 raft_consensus.cc:374] T 6869be9d2b8a48b8940932ce2158a165 P 2b229b3bf9b84796bb23853b0600c783 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:32.690104 5385 raft_consensus.cc:729] T 6869be9d2b8a48b8940932ce2158a165 P 2b229b3bf9b84796bb23853b0600c783 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b229b3bf9b84796bb23853b0600c783, State: Initialized, Role: FOLLOWER
I20260928 20:19:32.690224 5385 consensus_queue.cc:260] T 6869be9d2b8a48b8940932ce2158a165 P 2b229b3bf9b84796bb23853b0600c783 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b229b3bf9b84796bb23853b0600c783" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42595 } } peers { permanent_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34589 } } peers { permanent_uuid: "ae62d02eff7749979f0c98a8907c918a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38807 } }
I20260928 20:19:32.690465 5385 ts_tablet_manager.cc:1428] T 6869be9d2b8a48b8940932ce2158a165 P 2b229b3bf9b84796bb23853b0600c783: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 20:19:32.690543 5081 heartbeater.cc:503] Master 127.3.18.62:39065 was elected leader, sending a full tablet report...
W20260928 20:19:32.710453 5222 tablet.cc:2369] T 6869be9d2b8a48b8940932ce2158a165 P ba5667d185cb4f0e8d923ae42f7105bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 20:19:32.794834 5082 tablet.cc:2369] T 6869be9d2b8a48b8940932ce2158a165 P 2b229b3bf9b84796bb23853b0600c783: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 20:19:32.882440 5362 tablet.cc:2369] T 6869be9d2b8a48b8940932ce2158a165 P ae62d02eff7749979f0c98a8907c918a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 20:19:32.949939 5390 raft_consensus.cc:482] T 6869be9d2b8a48b8940932ce2158a165 P ae62d02eff7749979f0c98a8907c918a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:19:32.950139 5390 raft_consensus.cc:504] T 6869be9d2b8a48b8940932ce2158a165 P ae62d02eff7749979f0c98a8907c918a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b229b3bf9b84796bb23853b0600c783" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42595 } } peers { permanent_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34589 } } peers { permanent_uuid: "ae62d02eff7749979f0c98a8907c918a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38807 } }
I20260928 20:19:32.950533 5390 leader_election.cc:288] T 6869be9d2b8a48b8940932ce2158a165 P ae62d02eff7749979f0c98a8907c918a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b229b3bf9b84796bb23853b0600c783 (127.3.18.1:42595), ba5667d185cb4f0e8d923ae42f7105bf (127.3.18.2:34589)
I20260928 20:19:32.954241 5254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:32.950874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.2:34589 (local address 127.3.18.3:40423)
0928 20:19:32.950941 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:32.950954 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:32.951001 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:32.951292 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:32.951294 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:32.951296 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:19:32.951374 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:32.951376 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:32.952118 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:32.952119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:32.953050 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:32.953053 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:32.953858 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:32.953860 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:32.953977 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:32.953982 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 20:19:32.954046 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 20:19:32.954104 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":17}
I20260928 20:19:32.954243 5381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:32.950965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:34849 (local address 127.3.18.1:42595)
0928 20:19:32.951017 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:32.951019 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:32.951129 (+ 110us) server_negotiation.cc:407] Connection header received
0928 20:19:32.951195 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:32.951196 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:32.951209 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:32.951244 (+ 35us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:19:32.951451 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:32.951987 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:32.953164 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:32.953762 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:32.953841 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:32.953852 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:32.954083 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 20:19:32.954149 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 20:19:32.954246 5393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:32.950874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.1:42595 (local address 127.3.18.3:34849)
0928 20:19:32.951048 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:32.951057 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:32.951134 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:32.951264 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:32.951264 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:32.951279 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:19:32.951370 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:32.951372 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:32.952084 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:32.952085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:32.953047 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:32.953051 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:32.953905 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:32.953906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:32.953989 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:32.953992 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 20:19:32.954072 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 20:19:32.954122 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":60,"thread_start_us":97,"threads_started":1}
I20260928 20:19:32.954417 5382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:32.951059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:40423 (local address 127.3.18.2:34589)
0928 20:19:32.951118 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:32.951120 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:32.951129 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:19:32.951162 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:32.951163 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:32.951172 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:32.951255 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:19:32.951451 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:32.952026 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:32.953236 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:32.953762 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:32.953842 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:32.953852 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:32.954292 (+ 440us) server_negotiation.cc:299] Negotiation successful
0928 20:19:32.954342 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 20:19:32.954754 5175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6869be9d2b8a48b8940932ce2158a165" candidate_uuid: "ae62d02eff7749979f0c98a8907c918a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" is_pre_election: true
I20260928 20:19:32.954746 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6869be9d2b8a48b8940932ce2158a165" candidate_uuid: "ae62d02eff7749979f0c98a8907c918a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b229b3bf9b84796bb23853b0600c783" is_pre_election: true
I20260928 20:19:32.954906 5035 raft_consensus.cc:2454] T 6869be9d2b8a48b8940932ce2158a165 P 2b229b3bf9b84796bb23853b0600c783 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae62d02eff7749979f0c98a8907c918a in term 0.
I20260928 20:19:32.954907 5175 raft_consensus.cc:2454] T 6869be9d2b8a48b8940932ce2158a165 P ba5667d185cb4f0e8d923ae42f7105bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae62d02eff7749979f0c98a8907c918a in term 0.
I20260928 20:19:32.955317 5249 leader_election.cc:302] T 6869be9d2b8a48b8940932ce2158a165 P ae62d02eff7749979f0c98a8907c918a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b229b3bf9b84796bb23853b0600c783, ae62d02eff7749979f0c98a8907c918a; no voters:
I20260928 20:19:32.955572 5390 raft_consensus.cc:2781] T 6869be9d2b8a48b8940932ce2158a165 P ae62d02eff7749979f0c98a8907c918a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 20:19:32.955649 5390 raft_consensus.cc:482] T 6869be9d2b8a48b8940932ce2158a165 P ae62d02eff7749979f0c98a8907c918a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 20:19:32.955695 5390 raft_consensus.cc:3037] T 6869be9d2b8a48b8940932ce2158a165 P ae62d02eff7749979f0c98a8907c918a [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:32.956568 5390 raft_consensus.cc:504] T 6869be9d2b8a48b8940932ce2158a165 P ae62d02eff7749979f0c98a8907c918a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b229b3bf9b84796bb23853b0600c783" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42595 } } peers { permanent_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34589 } } peers { permanent_uuid: "ae62d02eff7749979f0c98a8907c918a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38807 } }
I20260928 20:19:32.956873 5390 leader_election.cc:288] T 6869be9d2b8a48b8940932ce2158a165 P ae62d02eff7749979f0c98a8907c918a [CANDIDATE]: Term 1 election: Requested vote from peers 2b229b3bf9b84796bb23853b0600c783 (127.3.18.1:42595), ba5667d185cb4f0e8d923ae42f7105bf (127.3.18.2:34589)
I20260928 20:19:32.956881 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6869be9d2b8a48b8940932ce2158a165" candidate_uuid: "ae62d02eff7749979f0c98a8907c918a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b229b3bf9b84796bb23853b0600c783"
I20260928 20:19:32.956967 5035 raft_consensus.cc:3037] T 6869be9d2b8a48b8940932ce2158a165 P 2b229b3bf9b84796bb23853b0600c783 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:32.957094 5175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6869be9d2b8a48b8940932ce2158a165" candidate_uuid: "ae62d02eff7749979f0c98a8907c918a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba5667d185cb4f0e8d923ae42f7105bf"
I20260928 20:19:32.957187 5175 raft_consensus.cc:3037] T 6869be9d2b8a48b8940932ce2158a165 P ba5667d185cb4f0e8d923ae42f7105bf [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:32.957975 5035 raft_consensus.cc:2454] T 6869be9d2b8a48b8940932ce2158a165 P 2b229b3bf9b84796bb23853b0600c783 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae62d02eff7749979f0c98a8907c918a in term 1.
I20260928 20:19:32.958128 5175 raft_consensus.cc:2454] T 6869be9d2b8a48b8940932ce2158a165 P ba5667d185cb4f0e8d923ae42f7105bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae62d02eff7749979f0c98a8907c918a in term 1.
I20260928 20:19:32.958168 5249 leader_election.cc:302] T 6869be9d2b8a48b8940932ce2158a165 P ae62d02eff7749979f0c98a8907c918a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b229b3bf9b84796bb23853b0600c783, ae62d02eff7749979f0c98a8907c918a; no voters:
I20260928 20:19:32.958312 5390 raft_consensus.cc:2781] T 6869be9d2b8a48b8940932ce2158a165 P ae62d02eff7749979f0c98a8907c918a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:19:32.958518 5390 raft_consensus.cc:686] T 6869be9d2b8a48b8940932ce2158a165 P ae62d02eff7749979f0c98a8907c918a [term 1 LEADER]: Becoming Leader. State: Replica: ae62d02eff7749979f0c98a8907c918a, State: Running, Role: LEADER
I20260928 20:19:32.958673 5390 consensus_queue.cc:237] T 6869be9d2b8a48b8940932ce2158a165 P ae62d02eff7749979f0c98a8907c918a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b229b3bf9b84796bb23853b0600c783" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42595 } } peers { permanent_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34589 } } peers { permanent_uuid: "ae62d02eff7749979f0c98a8907c918a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38807 } }
I20260928 20:19:32.959630 4877 catalog_manager.cc:5495] T 6869be9d2b8a48b8940932ce2158a165 P ae62d02eff7749979f0c98a8907c918a reported cstate change: term changed from 0 to 1, leader changed from <none> to ae62d02eff7749979f0c98a8907c918a (127.3.18.3). New cstate: current_term: 1 leader_uuid: "ae62d02eff7749979f0c98a8907c918a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b229b3bf9b84796bb23853b0600c783" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae62d02eff7749979f0c98a8907c918a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38807 } health_report { overall_health: HEALTHY } } }
Sep 28 20:19:33 dist-test-slave-b83v krb5kdc[4833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626772, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.62@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:19:33.056015 4932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:33.048632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60432 (local address 127.3.18.62:39065)
0928 20:19:33.048724 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:33.048726 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:33.048892 (+ 166us) server_negotiation.cc:407] Connection header received
0928 20:19:33.048963 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:33.048964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:33.048973 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:33.049027 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:33.049597 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:33.050094 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:33.050874 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:33.051061 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:33.051116 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:33.053348 (+ 2232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:33.053349 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:33.053350 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:33.053356 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:33.054812 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:33.055227 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:33.055228 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:33.055228 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:33.055288 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:33.055456 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:33.055457 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:33.055457 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:33.055573 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:33.055623 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:33.055865 (+ 242us) server_negotiation.cc:299] Negotiation successful
0928 20:19:33.055946 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 20:19:33.056113 5401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:33.048560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:39065 (local address 127.0.0.1:60432)
0928 20:19:33.048801 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:33.048812 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:33.048891 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:33.049069 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:33.049071 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:33.049459 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:33.049509 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:33.049512 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:33.050174 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:33.050176 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:33.050796 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:33.050798 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:33.051159 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:33.051160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:33.051214 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:33.051728 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:19:33.051733 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:33.053258 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:33.054900 (+ 1642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:33.054906 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:33.054909 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:33.055110 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:33.055346 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:33.055346 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:33.055347 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:33.055405 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:33.055652 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:33.055659 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:33.055712 (+ 53us) client_negotiation.cc:769] Sending connection context
0928 20:19:33.055845 (+ 133us) client_negotiation.cc:240] Negotiation successful
0928 20:19:33.056025 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":68,"thread_start_us":151,"threads_started":1}
I20260928 20:19:33.058434 4877 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:60432:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 20:19:33.058678 4877 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 20:19:33.060387 5015 tablet_service.cc:1467] Processing CreateTablet for tablet 5be2e6e258b34e6ebd6249b57adc3abc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3a7c2981643547e7a10c617672127838]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 20:19:33.060405 5155 tablet_service.cc:1467] Processing CreateTablet for tablet 5be2e6e258b34e6ebd6249b57adc3abc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3a7c2981643547e7a10c617672127838]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 20:19:33.060566 5015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5be2e6e258b34e6ebd6249b57adc3abc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:33.060567 5155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5be2e6e258b34e6ebd6249b57adc3abc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:33.060622 5295 tablet_service.cc:1467] Processing CreateTablet for tablet 5be2e6e258b34e6ebd6249b57adc3abc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3a7c2981643547e7a10c617672127838]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 20:19:33.060787 5295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5be2e6e258b34e6ebd6249b57adc3abc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:33.061905 5384 tablet_bootstrap.cc:492] T 5be2e6e258b34e6ebd6249b57adc3abc P ae62d02eff7749979f0c98a8907c918a: Bootstrap starting.
I20260928 20:19:33.062119 5386 tablet_bootstrap.cc:492] T 5be2e6e258b34e6ebd6249b57adc3abc P ba5667d185cb4f0e8d923ae42f7105bf: Bootstrap starting.
I20260928 20:19:33.062129 5385 tablet_bootstrap.cc:492] T 5be2e6e258b34e6ebd6249b57adc3abc P 2b229b3bf9b84796bb23853b0600c783: Bootstrap starting.
I20260928 20:19:33.062381 5384 tablet_bootstrap.cc:654] T 5be2e6e258b34e6ebd6249b57adc3abc P ae62d02eff7749979f0c98a8907c918a: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:33.062487 5386 tablet_bootstrap.cc:654] T 5be2e6e258b34e6ebd6249b57adc3abc P ba5667d185cb4f0e8d923ae42f7105bf: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:33.062516 5385 tablet_bootstrap.cc:654] T 5be2e6e258b34e6ebd6249b57adc3abc P 2b229b3bf9b84796bb23853b0600c783: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:33.063092 5386 tablet_bootstrap.cc:492] T 5be2e6e258b34e6ebd6249b57adc3abc P ba5667d185cb4f0e8d923ae42f7105bf: No bootstrap required, opened a new log
I20260928 20:19:33.063108 5384 tablet_bootstrap.cc:492] T 5be2e6e258b34e6ebd6249b57adc3abc P ae62d02eff7749979f0c98a8907c918a: No bootstrap required, opened a new log
I20260928 20:19:33.063150 5386 ts_tablet_manager.cc:1397] T 5be2e6e258b34e6ebd6249b57adc3abc P ba5667d185cb4f0e8d923ae42f7105bf: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 20:19:33.063150 5384 ts_tablet_manager.cc:1397] T 5be2e6e258b34e6ebd6249b57adc3abc P ae62d02eff7749979f0c98a8907c918a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 20:19:33.063215 5385 tablet_bootstrap.cc:492] T 5be2e6e258b34e6ebd6249b57adc3abc P 2b229b3bf9b84796bb23853b0600c783: No bootstrap required, opened a new log
I20260928 20:19:33.063265 5385 ts_tablet_manager.cc:1397] T 5be2e6e258b34e6ebd6249b57adc3abc P 2b229b3bf9b84796bb23853b0600c783: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 20:19:33.063304 5386 raft_consensus.cc:348] T 5be2e6e258b34e6ebd6249b57adc3abc P ba5667d185cb4f0e8d923ae42f7105bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b229b3bf9b84796bb23853b0600c783" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42595 } } peers { permanent_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34589 } } peers { permanent_uuid: "ae62d02eff7749979f0c98a8907c918a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38807 } }
I20260928 20:19:33.063304 5384 raft_consensus.cc:348] T 5be2e6e258b34e6ebd6249b57adc3abc P ae62d02eff7749979f0c98a8907c918a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b229b3bf9b84796bb23853b0600c783" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42595 } } peers { permanent_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34589 } } peers { permanent_uuid: "ae62d02eff7749979f0c98a8907c918a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38807 } }
I20260928 20:19:33.063385 5386 raft_consensus.cc:374] T 5be2e6e258b34e6ebd6249b57adc3abc P ba5667d185cb4f0e8d923ae42f7105bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:33.063385 5384 raft_consensus.cc:374] T 5be2e6e258b34e6ebd6249b57adc3abc P ae62d02eff7749979f0c98a8907c918a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:33.063412 5384 raft_consensus.cc:729] T 5be2e6e258b34e6ebd6249b57adc3abc P ae62d02eff7749979f0c98a8907c918a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae62d02eff7749979f0c98a8907c918a, State: Initialized, Role: FOLLOWER
I20260928 20:19:33.063412 5386 raft_consensus.cc:729] T 5be2e6e258b34e6ebd6249b57adc3abc P ba5667d185cb4f0e8d923ae42f7105bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba5667d185cb4f0e8d923ae42f7105bf, State: Initialized, Role: FOLLOWER
I20260928 20:19:33.063431 5385 raft_consensus.cc:348] T 5be2e6e258b34e6ebd6249b57adc3abc P 2b229b3bf9b84796bb23853b0600c783 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b229b3bf9b84796bb23853b0600c783" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42595 } } peers { permanent_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34589 } } peers { permanent_uuid: "ae62d02eff7749979f0c98a8907c918a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38807 } }
I20260928 20:19:33.063457 5386 consensus_queue.cc:260] T 5be2e6e258b34e6ebd6249b57adc3abc P ba5667d185cb4f0e8d923ae42f7105bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b229b3bf9b84796bb23853b0600c783" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42595 } } peers { permanent_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34589 } } peers { permanent_uuid: "ae62d02eff7749979f0c98a8907c918a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38807 } }
I20260928 20:19:33.063454 5384 consensus_queue.cc:260] T 5be2e6e258b34e6ebd6249b57adc3abc P ae62d02eff7749979f0c98a8907c918a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b229b3bf9b84796bb23853b0600c783" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42595 } } peers { permanent_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34589 } } peers { permanent_uuid: "ae62d02eff7749979f0c98a8907c918a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38807 } }
I20260928 20:19:33.063493 5385 raft_consensus.cc:374] T 5be2e6e258b34e6ebd6249b57adc3abc P 2b229b3bf9b84796bb23853b0600c783 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:33.063512 5385 raft_consensus.cc:729] T 5be2e6e258b34e6ebd6249b57adc3abc P 2b229b3bf9b84796bb23853b0600c783 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b229b3bf9b84796bb23853b0600c783, State: Initialized, Role: FOLLOWER
I20260928 20:19:33.063555 5384 ts_tablet_manager.cc:1428] T 5be2e6e258b34e6ebd6249b57adc3abc P ae62d02eff7749979f0c98a8907c918a: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 20:19:33.063560 5386 ts_tablet_manager.cc:1428] T 5be2e6e258b34e6ebd6249b57adc3abc P ba5667d185cb4f0e8d923ae42f7105bf: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:33.063556 5385 consensus_queue.cc:260] T 5be2e6e258b34e6ebd6249b57adc3abc P 2b229b3bf9b84796bb23853b0600c783 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b229b3bf9b84796bb23853b0600c783" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42595 } } peers { permanent_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34589 } } peers { permanent_uuid: "ae62d02eff7749979f0c98a8907c918a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38807 } }
I20260928 20:19:33.063671 5385 ts_tablet_manager.cc:1428] T 5be2e6e258b34e6ebd6249b57adc3abc P 2b229b3bf9b84796bb23853b0600c783: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 20:19:33.063715 5390 tablet_replica.cc:439] T 5be2e6e258b34e6ebd6249b57adc3abc P ae62d02eff7749979f0c98a8907c918a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b229b3bf9b84796bb23853b0600c783" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42595 } } peers { permanent_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34589 } } peers { permanent_uuid: "ae62d02eff7749979f0c98a8907c918a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38807 } } }
I20260928 20:19:33.063812 5390 tablet_replica.cc:442] T 5be2e6e258b34e6ebd6249b57adc3abc P ae62d02eff7749979f0c98a8907c918a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 20:19:33.063884 5392 tablet_replica.cc:439] T 5be2e6e258b34e6ebd6249b57adc3abc P 2b229b3bf9b84796bb23853b0600c783: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b229b3bf9b84796bb23853b0600c783" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42595 } } peers { permanent_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34589 } } peers { permanent_uuid: "ae62d02eff7749979f0c98a8907c918a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38807 } } }
I20260928 20:19:33.063958 5392 tablet_replica.cc:442] T 5be2e6e258b34e6ebd6249b57adc3abc P 2b229b3bf9b84796bb23853b0600c783: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 20:19:33.063951 5391 tablet_replica.cc:439] T 5be2e6e258b34e6ebd6249b57adc3abc P ba5667d185cb4f0e8d923ae42f7105bf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b229b3bf9b84796bb23853b0600c783" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42595 } } peers { permanent_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34589 } } peers { permanent_uuid: "ae62d02eff7749979f0c98a8907c918a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38807 } } }
I20260928 20:19:33.064023 5391 tablet_replica.cc:442] T 5be2e6e258b34e6ebd6249b57adc3abc P ba5667d185cb4f0e8d923ae42f7105bf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 20:19:33.254446 5390 raft_consensus.cc:482] T 5be2e6e258b34e6ebd6249b57adc3abc P ae62d02eff7749979f0c98a8907c918a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:19:33.254558 5390 raft_consensus.cc:504] T 5be2e6e258b34e6ebd6249b57adc3abc P ae62d02eff7749979f0c98a8907c918a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b229b3bf9b84796bb23853b0600c783" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42595 } } peers { permanent_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34589 } } peers { permanent_uuid: "ae62d02eff7749979f0c98a8907c918a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38807 } }
I20260928 20:19:33.254745 5390 leader_election.cc:288] T 5be2e6e258b34e6ebd6249b57adc3abc P ae62d02eff7749979f0c98a8907c918a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b229b3bf9b84796bb23853b0600c783 (127.3.18.1:42595), ba5667d185cb4f0e8d923ae42f7105bf (127.3.18.2:34589)
I20260928 20:19:33.255007 5175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5be2e6e258b34e6ebd6249b57adc3abc" candidate_uuid: "ae62d02eff7749979f0c98a8907c918a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" is_pre_election: true
I20260928 20:19:33.255007 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5be2e6e258b34e6ebd6249b57adc3abc" candidate_uuid: "ae62d02eff7749979f0c98a8907c918a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b229b3bf9b84796bb23853b0600c783" is_pre_election: true
I20260928 20:19:33.255121 5175 raft_consensus.cc:2454] T 5be2e6e258b34e6ebd6249b57adc3abc P ba5667d185cb4f0e8d923ae42f7105bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae62d02eff7749979f0c98a8907c918a in term 0.
I20260928 20:19:33.255122 5035 raft_consensus.cc:2454] T 5be2e6e258b34e6ebd6249b57adc3abc P 2b229b3bf9b84796bb23853b0600c783 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae62d02eff7749979f0c98a8907c918a in term 0.
I20260928 20:19:33.255293 5249 leader_election.cc:302] T 5be2e6e258b34e6ebd6249b57adc3abc P ae62d02eff7749979f0c98a8907c918a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b229b3bf9b84796bb23853b0600c783, ae62d02eff7749979f0c98a8907c918a; no voters:
I20260928 20:19:33.255415 5390 raft_consensus.cc:2781] T 5be2e6e258b34e6ebd6249b57adc3abc P ae62d02eff7749979f0c98a8907c918a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 20:19:33.255473 5390 raft_consensus.cc:482] T 5be2e6e258b34e6ebd6249b57adc3abc P ae62d02eff7749979f0c98a8907c918a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 20:19:33.255503 5390 raft_consensus.cc:3037] T 5be2e6e258b34e6ebd6249b57adc3abc P ae62d02eff7749979f0c98a8907c918a [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:33.256321 5390 raft_consensus.cc:504] T 5be2e6e258b34e6ebd6249b57adc3abc P ae62d02eff7749979f0c98a8907c918a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b229b3bf9b84796bb23853b0600c783" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42595 } } peers { permanent_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34589 } } peers { permanent_uuid: "ae62d02eff7749979f0c98a8907c918a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38807 } }
I20260928 20:19:33.256494 5390 leader_election.cc:288] T 5be2e6e258b34e6ebd6249b57adc3abc P ae62d02eff7749979f0c98a8907c918a [CANDIDATE]: Term 1 election: Requested vote from peers 2b229b3bf9b84796bb23853b0600c783 (127.3.18.1:42595), ba5667d185cb4f0e8d923ae42f7105bf (127.3.18.2:34589)
I20260928 20:19:33.256695 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5be2e6e258b34e6ebd6249b57adc3abc" candidate_uuid: "ae62d02eff7749979f0c98a8907c918a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b229b3bf9b84796bb23853b0600c783"
I20260928 20:19:33.256698 5175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5be2e6e258b34e6ebd6249b57adc3abc" candidate_uuid: "ae62d02eff7749979f0c98a8907c918a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba5667d185cb4f0e8d923ae42f7105bf"
I20260928 20:19:33.256803 5175 raft_consensus.cc:3037] T 5be2e6e258b34e6ebd6249b57adc3abc P ba5667d185cb4f0e8d923ae42f7105bf [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:33.256806 5035 raft_consensus.cc:3037] T 5be2e6e258b34e6ebd6249b57adc3abc P 2b229b3bf9b84796bb23853b0600c783 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:33.257488 5175 raft_consensus.cc:2454] T 5be2e6e258b34e6ebd6249b57adc3abc P ba5667d185cb4f0e8d923ae42f7105bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae62d02eff7749979f0c98a8907c918a in term 1.
I20260928 20:19:33.257581 5035 raft_consensus.cc:2454] T 5be2e6e258b34e6ebd6249b57adc3abc P 2b229b3bf9b84796bb23853b0600c783 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae62d02eff7749979f0c98a8907c918a in term 1.
I20260928 20:19:33.257678 5248 leader_election.cc:302] T 5be2e6e258b34e6ebd6249b57adc3abc P ae62d02eff7749979f0c98a8907c918a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae62d02eff7749979f0c98a8907c918a, ba5667d185cb4f0e8d923ae42f7105bf; no voters:
I20260928 20:19:33.257771 5390 raft_consensus.cc:2781] T 5be2e6e258b34e6ebd6249b57adc3abc P ae62d02eff7749979f0c98a8907c918a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:19:33.257835 5390 raft_consensus.cc:686] T 5be2e6e258b34e6ebd6249b57adc3abc P ae62d02eff7749979f0c98a8907c918a [term 1 LEADER]: Becoming Leader. State: Replica: ae62d02eff7749979f0c98a8907c918a, State: Running, Role: LEADER
I20260928 20:19:33.257894 5390 consensus_queue.cc:237] T 5be2e6e258b34e6ebd6249b57adc3abc P ae62d02eff7749979f0c98a8907c918a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b229b3bf9b84796bb23853b0600c783" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42595 } } peers { permanent_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34589 } } peers { permanent_uuid: "ae62d02eff7749979f0c98a8907c918a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38807 } }
I20260928 20:19:33.258213 5394 tablet_replica.cc:439] T 5be2e6e258b34e6ebd6249b57adc3abc P ae62d02eff7749979f0c98a8907c918a: TxnStatusTablet state changed. Reason: New leader ae62d02eff7749979f0c98a8907c918a. Latest consensus state: current_term: 1 leader_uuid: "ae62d02eff7749979f0c98a8907c918a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b229b3bf9b84796bb23853b0600c783" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42595 } } peers { permanent_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34589 } } peers { permanent_uuid: "ae62d02eff7749979f0c98a8907c918a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38807 } } }
I20260928 20:19:33.258303 5394 tablet_replica.cc:442] T 5be2e6e258b34e6ebd6249b57adc3abc P ae62d02eff7749979f0c98a8907c918a: This TxnStatusTablet replica's current role is: LEADER
I20260928 20:19:33.258479 5406 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 20:19:33.258621 5406 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 20:19:33.258605 4878 catalog_manager.cc:5495] T 5be2e6e258b34e6ebd6249b57adc3abc P ae62d02eff7749979f0c98a8907c918a reported cstate change: term changed from 0 to 1, leader changed from <none> to ae62d02eff7749979f0c98a8907c918a (127.3.18.3). New cstate: current_term: 1 leader_uuid: "ae62d02eff7749979f0c98a8907c918a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b229b3bf9b84796bb23853b0600c783" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae62d02eff7749979f0c98a8907c918a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38807 } health_report { overall_health: HEALTHY } } }
I20260928 20:19:33.295393 5409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:33.292253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48112 (local address 127.3.18.3:38807)
0928 20:19:33.292359 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:33.292361 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:33.292370 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:19:33.292418 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:33.292419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:33.292436 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:33.292505 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:19:33.292675 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:33.293178 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:33.294301 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:33.294658 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:33.294741 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:33.294968 (+ 227us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:19:33.295110 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:19:33.295203 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:33.295250 (+ 47us) server_negotiation.cc:299] Negotiation successful
0928 20:19:33.295275 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":65,"threads_started":1}
I20260928 20:19:33.295406 5401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:33.292168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.3:38807 (local address 127.0.0.1:48112)
0928 20:19:33.292279 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:33.292291 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:33.292346 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:33.292528 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:33.292528 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:33.292530 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 20:19:33.292588 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:33.292591 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:33.293254 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:33.293255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:33.294201 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:33.294205 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:33.294752 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:33.294754 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:33.294824 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:33.294847 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 20:19:33.295187 (+ 340us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 20:19:33.295195 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 20:19:33.295247 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 20:19:33.295301 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260928 20:19:33.297371 5035 raft_consensus.cc:1261] T 5be2e6e258b34e6ebd6249b57adc3abc P 2b229b3bf9b84796bb23853b0600c783 [term 1 FOLLOWER]: Refusing update from remote peer ae62d02eff7749979f0c98a8907c918a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:19:33.297422 5175 raft_consensus.cc:1261] T 5be2e6e258b34e6ebd6249b57adc3abc P ba5667d185cb4f0e8d923ae42f7105bf [term 1 FOLLOWER]: Refusing update from remote peer ae62d02eff7749979f0c98a8907c918a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:19:33.297719 5394 consensus_queue.cc:1035] T 5be2e6e258b34e6ebd6249b57adc3abc P ae62d02eff7749979f0c98a8907c918a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b229b3bf9b84796bb23853b0600c783" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:19:33.297869 5390 consensus_queue.cc:1035] T 5be2e6e258b34e6ebd6249b57adc3abc P ae62d02eff7749979f0c98a8907c918a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:19:33.299118 5391 tablet_replica.cc:439] T 5be2e6e258b34e6ebd6249b57adc3abc P ba5667d185cb4f0e8d923ae42f7105bf: TxnStatusTablet state changed. Reason: New leader ae62d02eff7749979f0c98a8907c918a. Latest consensus state: current_term: 1 leader_uuid: "ae62d02eff7749979f0c98a8907c918a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b229b3bf9b84796bb23853b0600c783" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42595 } } peers { permanent_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34589 } } peers { permanent_uuid: "ae62d02eff7749979f0c98a8907c918a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38807 } } }
I20260928 20:19:33.299214 5391 tablet_replica.cc:442] T 5be2e6e258b34e6ebd6249b57adc3abc P ba5667d185cb4f0e8d923ae42f7105bf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 20:19:33.299131 5392 tablet_replica.cc:439] T 5be2e6e258b34e6ebd6249b57adc3abc P 2b229b3bf9b84796bb23853b0600c783: TxnStatusTablet state changed. Reason: New leader ae62d02eff7749979f0c98a8907c918a. Latest consensus state: current_term: 1 leader_uuid: "ae62d02eff7749979f0c98a8907c918a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b229b3bf9b84796bb23853b0600c783" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42595 } } peers { permanent_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34589 } } peers { permanent_uuid: "ae62d02eff7749979f0c98a8907c918a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38807 } } }
I20260928 20:19:33.299283 5392 tablet_replica.cc:442] T 5be2e6e258b34e6ebd6249b57adc3abc P 2b229b3bf9b84796bb23853b0600c783: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 20:19:33.300489 5390 tablet_replica.cc:439] T 5be2e6e258b34e6ebd6249b57adc3abc P ae62d02eff7749979f0c98a8907c918a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ae62d02eff7749979f0c98a8907c918a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b229b3bf9b84796bb23853b0600c783" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42595 } } peers { permanent_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34589 } } peers { permanent_uuid: "ae62d02eff7749979f0c98a8907c918a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38807 } } }
I20260928 20:19:33.300583 5390 tablet_replica.cc:442] T 5be2e6e258b34e6ebd6249b57adc3abc P ae62d02eff7749979f0c98a8907c918a: This TxnStatusTablet replica's current role is: LEADER
I20260928 20:19:33.300796 5390 tablet_replica.cc:439] T 5be2e6e258b34e6ebd6249b57adc3abc P ae62d02eff7749979f0c98a8907c918a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ae62d02eff7749979f0c98a8907c918a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b229b3bf9b84796bb23853b0600c783" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42595 } } peers { permanent_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34589 } } peers { permanent_uuid: "ae62d02eff7749979f0c98a8907c918a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38807 } } }
I20260928 20:19:33.300889 5390 tablet_replica.cc:442] T 5be2e6e258b34e6ebd6249b57adc3abc P ae62d02eff7749979f0c98a8907c918a: This TxnStatusTablet replica's current role is: LEADER
I20260928 20:19:33.301071 5391 tablet_replica.cc:439] T 5be2e6e258b34e6ebd6249b57adc3abc P ba5667d185cb4f0e8d923ae42f7105bf: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ae62d02eff7749979f0c98a8907c918a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b229b3bf9b84796bb23853b0600c783" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42595 } } peers { permanent_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34589 } } peers { permanent_uuid: "ae62d02eff7749979f0c98a8907c918a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38807 } } }
I20260928 20:19:33.301163 5391 tablet_replica.cc:442] T 5be2e6e258b34e6ebd6249b57adc3abc P ba5667d185cb4f0e8d923ae42f7105bf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 20:19:33.301173 5392 tablet_replica.cc:439] T 5be2e6e258b34e6ebd6249b57adc3abc P 2b229b3bf9b84796bb23853b0600c783: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ae62d02eff7749979f0c98a8907c918a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b229b3bf9b84796bb23853b0600c783" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42595 } } peers { permanent_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34589 } } peers { permanent_uuid: "ae62d02eff7749979f0c98a8907c918a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38807 } } }
I20260928 20:19:33.301249 5392 tablet_replica.cc:442] T 5be2e6e258b34e6ebd6249b57adc3abc P 2b229b3bf9b84796bb23853b0600c783: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 20:19:33.301626 5411 mvcc.cc:205] Tried to move back new op lower bound from 7334407263423561728 to 7334407263264755712. Current Snapshot: MvccSnapshot[applied={T|T < 7334407263423561728 or (T in {7334407263423561728})}]
I20260928 20:19:33.302129 5412 mvcc.cc:205] Tried to move back new op lower bound from 7334407263423561728 to 7334407263264755712. Current Snapshot: MvccSnapshot[applied={T|T < 7334407263423561728}]
I20260928 20:19:33.306784 5409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:33.304034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48120 (local address 127.3.18.3:38807)
0928 20:19:33.304078 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:33.304080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:33.304235 (+ 155us) server_negotiation.cc:407] Connection header received
0928 20:19:33.304263 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:33.304264 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:33.304274 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:33.304342 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:19:33.304425 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:33.305135 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:33.305958 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:33.306270 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:33.306344 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:33.306493 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:19:33.306528 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:19:33.306580 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:33.306658 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 20:19:33.306698 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 20:19:33.312222 5409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:33.309413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:38571 (local address 127.3.18.3:38807)
0928 20:19:33.309458 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:33.309460 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:33.309606 (+ 146us) server_negotiation.cc:407] Connection header received
0928 20:19:33.309658 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:33.309659 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:33.309668 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:33.309722 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:19:33.309876 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:33.310353 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:33.311458 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:33.311630 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:33.311688 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:33.311950 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:19:33.311979 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:19:33.312022 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:33.312082 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 20:19:33.312142 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 20:19:33.312229 5422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:33.309322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.3:38807 (local address 127.3.18.3:38571)
0928 20:19:33.309524 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:33.309533 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:33.309596 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:33.309728 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:33.309729 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:33.309731 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 20:19:33.309816 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:33.309817 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:33.310428 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:33.310429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:33.311375 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:33.311377 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:33.311705 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:33.311706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:33.311826 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:33.311841 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 20:19:33.312037 (+ 196us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 20:19:33.312038 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 20:19:33.312090 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 20:19:33.312131 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":103,"threads_started":1}
I20260928 20:19:33.314473 5035 raft_consensus.cc:1261] T 6869be9d2b8a48b8940932ce2158a165 P 2b229b3bf9b84796bb23853b0600c783 [term 1 FOLLOWER]: Refusing update from remote peer ae62d02eff7749979f0c98a8907c918a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:19:33.314559 5175 raft_consensus.cc:1261] T 6869be9d2b8a48b8940932ce2158a165 P ba5667d185cb4f0e8d923ae42f7105bf [term 1 FOLLOWER]: Refusing update from remote peer ae62d02eff7749979f0c98a8907c918a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:19:33.314760 5394 consensus_queue.cc:1035] T 6869be9d2b8a48b8940932ce2158a165 P ae62d02eff7749979f0c98a8907c918a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b229b3bf9b84796bb23853b0600c783" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:19:33.314854 5390 consensus_queue.cc:1035] T 6869be9d2b8a48b8940932ce2158a165 P ae62d02eff7749979f0c98a8907c918a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:19:33.316174 5410 mvcc.cc:205] Tried to move back new op lower bound from 7334407263494488064 to 7334407262039355392. Current Snapshot: MvccSnapshot[applied={T|T < 7334407263494488064}]
I20260928 20:19:33.328663 4877 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:19:33.328797 4877 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:19:33.329586 4877 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cddefad8370040909095f66cc0817b0e: Sending DeleteTablet for 3 replicas of tablet 6869be9d2b8a48b8940932ce2158a165
I20260928 20:19:33.329916 5294 tablet_service.cc:1514] Processing DeleteTablet for tablet 6869be9d2b8a48b8940932ce2158a165 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:19:33 UTC) from {username='kudu', principal='kudu/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:48108
I20260928 20:19:33.329950 5015 tablet_service.cc:1514] Processing DeleteTablet for tablet 6869be9d2b8a48b8940932ce2158a165 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:19:33 UTC) from {username='kudu', principal='kudu/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:42410
I20260928 20:19:33.330001 5155 tablet_service.cc:1514] Processing DeleteTablet for tablet 6869be9d2b8a48b8940932ce2158a165 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:19:33 UTC) from {username='kudu', principal='kudu/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:49984
I20260928 20:19:33.330206 5427 tablet_replica.cc:330] T 6869be9d2b8a48b8940932ce2158a165 P ae62d02eff7749979f0c98a8907c918a: stopping tablet replica
I20260928 20:19:33.330331 5427 raft_consensus.cc:2229] T 6869be9d2b8a48b8940932ce2158a165 P ae62d02eff7749979f0c98a8907c918a [term 1 LEADER]: Raft consensus shutting down.
I20260928 20:19:33.330515 5427 raft_consensus.cc:2258] T 6869be9d2b8a48b8940932ce2158a165 P ae62d02eff7749979f0c98a8907c918a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:19:33.330482 4877 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60422:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 20:19:33.330636 4877 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 20:19:33.331005 5427 ts_tablet_manager.cc:1905] T 6869be9d2b8a48b8940932ce2158a165 P ae62d02eff7749979f0c98a8907c918a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:19:33.331216 5428 tablet_replica.cc:330] T 6869be9d2b8a48b8940932ce2158a165 P ba5667d185cb4f0e8d923ae42f7105bf: stopping tablet replica
I20260928 20:19:33.331372 5429 tablet_replica.cc:330] T 6869be9d2b8a48b8940932ce2158a165 P 2b229b3bf9b84796bb23853b0600c783: stopping tablet replica
I20260928 20:19:33.331511 5429 raft_consensus.cc:2229] T 6869be9d2b8a48b8940932ce2158a165 P 2b229b3bf9b84796bb23853b0600c783 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:19:33.331597 5429 raft_consensus.cc:2258] T 6869be9d2b8a48b8940932ce2158a165 P 2b229b3bf9b84796bb23853b0600c783 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:19:33.332073 5429 ts_tablet_manager.cc:1905] T 6869be9d2b8a48b8940932ce2158a165 P 2b229b3bf9b84796bb23853b0600c783: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:19:33.332242 5428 raft_consensus.cc:2229] T 6869be9d2b8a48b8940932ce2158a165 P ba5667d185cb4f0e8d923ae42f7105bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:19:33.332397 5155 tablet_service.cc:1467] Processing CreateTablet for tablet 9cea7e93054d4498b57f0bfe4b312d3f (DEFAULT_TABLE table=test-table [id=fe51b6f3788a4006a137ad5696559a3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:19:33.332510 5428 raft_consensus.cc:2258] T 6869be9d2b8a48b8940932ce2158a165 P ba5667d185cb4f0e8d923ae42f7105bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:19:33.332554 5155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cea7e93054d4498b57f0bfe4b312d3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:33.332639 5294 tablet_service.cc:1467] Processing CreateTablet for tablet 9cea7e93054d4498b57f0bfe4b312d3f (DEFAULT_TABLE table=test-table [id=fe51b6f3788a4006a137ad5696559a3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:19:33.332746 5294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cea7e93054d4498b57f0bfe4b312d3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:33.332865 5428 ts_tablet_manager.cc:1905] T 6869be9d2b8a48b8940932ce2158a165 P ba5667d185cb4f0e8d923ae42f7105bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:19:33.333263 5427 ts_tablet_manager.cc:1918] T 6869be9d2b8a48b8940932ce2158a165 P ae62d02eff7749979f0c98a8907c918a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 20:19:33.333330 5427 log.cc:1196] T 6869be9d2b8a48b8940932ce2158a165 P ae62d02eff7749979f0c98a8907c918a: Deleting WAL directory at /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/ts-2/wal/wals/6869be9d2b8a48b8940932ce2158a165
I20260928 20:19:33.333621 5427 ts_tablet_manager.cc:1939] T 6869be9d2b8a48b8940932ce2158a165 P ae62d02eff7749979f0c98a8907c918a: Deleting consensus metadata
I20260928 20:19:33.333990 5386 tablet_bootstrap.cc:492] T 9cea7e93054d4498b57f0bfe4b312d3f P ba5667d185cb4f0e8d923ae42f7105bf: Bootstrap starting.
I20260928 20:19:33.334224 5384 tablet_bootstrap.cc:492] T 9cea7e93054d4498b57f0bfe4b312d3f P ae62d02eff7749979f0c98a8907c918a: Bootstrap starting.
I20260928 20:19:33.334383 5386 tablet_bootstrap.cc:654] T 9cea7e93054d4498b57f0bfe4b312d3f P ba5667d185cb4f0e8d923ae42f7105bf: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:33.334415 5428 ts_tablet_manager.cc:1918] T 6869be9d2b8a48b8940932ce2158a165 P ba5667d185cb4f0e8d923ae42f7105bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 20:19:33.334477 5428 log.cc:1196] T 6869be9d2b8a48b8940932ce2158a165 P ba5667d185cb4f0e8d923ae42f7105bf: Deleting WAL directory at /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/ts-1/wal/wals/6869be9d2b8a48b8940932ce2158a165
I20260928 20:19:33.334656 5384 tablet_bootstrap.cc:654] T 9cea7e93054d4498b57f0bfe4b312d3f P ae62d02eff7749979f0c98a8907c918a: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:33.334735 5428 ts_tablet_manager.cc:1939] T 6869be9d2b8a48b8940932ce2158a165 P ba5667d185cb4f0e8d923ae42f7105bf: Deleting consensus metadata
I20260928 20:19:33.332438 5015 tablet_service.cc:1467] Processing CreateTablet for tablet 9cea7e93054d4498b57f0bfe4b312d3f (DEFAULT_TABLE table=test-table [id=fe51b6f3788a4006a137ad5696559a3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:19:33.334959 5015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cea7e93054d4498b57f0bfe4b312d3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:33.335054 4863 catalog_manager.cc:4841] TS ae62d02eff7749979f0c98a8907c918a (127.3.18.3:38807): tablet 6869be9d2b8a48b8940932ce2158a165 (table test-table [id=1209d70dcf9b479091536365d16bcfed]) successfully deleted
I20260928 20:19:33.335300 5384 tablet_bootstrap.cc:492] T 9cea7e93054d4498b57f0bfe4b312d3f P ae62d02eff7749979f0c98a8907c918a: No bootstrap required, opened a new log
I20260928 20:19:33.335304 5386 tablet_bootstrap.cc:492] T 9cea7e93054d4498b57f0bfe4b312d3f P ba5667d185cb4f0e8d923ae42f7105bf: No bootstrap required, opened a new log
I20260928 20:19:33.335347 5384 ts_tablet_manager.cc:1397] T 9cea7e93054d4498b57f0bfe4b312d3f P ae62d02eff7749979f0c98a8907c918a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 20:19:33.335347 5386 ts_tablet_manager.cc:1397] T 9cea7e93054d4498b57f0bfe4b312d3f P ba5667d185cb4f0e8d923ae42f7105bf: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 20:19:33.335485 5384 raft_consensus.cc:348] T 9cea7e93054d4498b57f0bfe4b312d3f P ae62d02eff7749979f0c98a8907c918a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b229b3bf9b84796bb23853b0600c783" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42595 } } peers { permanent_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34589 } } peers { permanent_uuid: "ae62d02eff7749979f0c98a8907c918a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38807 } }
I20260928 20:19:33.335494 5386 raft_consensus.cc:348] T 9cea7e93054d4498b57f0bfe4b312d3f P ba5667d185cb4f0e8d923ae42f7105bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b229b3bf9b84796bb23853b0600c783" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42595 } } peers { permanent_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34589 } } peers { permanent_uuid: "ae62d02eff7749979f0c98a8907c918a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38807 } }
I20260928 20:19:33.335554 5386 raft_consensus.cc:374] T 9cea7e93054d4498b57f0bfe4b312d3f P ba5667d185cb4f0e8d923ae42f7105bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:33.335552 5384 raft_consensus.cc:374] T 9cea7e93054d4498b57f0bfe4b312d3f P ae62d02eff7749979f0c98a8907c918a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:33.335577 5384 raft_consensus.cc:729] T 9cea7e93054d4498b57f0bfe4b312d3f P ae62d02eff7749979f0c98a8907c918a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae62d02eff7749979f0c98a8907c918a, State: Initialized, Role: FOLLOWER
I20260928 20:19:33.335574 5386 raft_consensus.cc:729] T 9cea7e93054d4498b57f0bfe4b312d3f P ba5667d185cb4f0e8d923ae42f7105bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba5667d185cb4f0e8d923ae42f7105bf, State: Initialized, Role: FOLLOWER
I20260928 20:19:33.335615 5386 consensus_queue.cc:260] T 9cea7e93054d4498b57f0bfe4b312d3f P ba5667d185cb4f0e8d923ae42f7105bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b229b3bf9b84796bb23853b0600c783" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42595 } } peers { permanent_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34589 } } peers { permanent_uuid: "ae62d02eff7749979f0c98a8907c918a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38807 } }
I20260928 20:19:33.335654 5384 consensus_queue.cc:260] T 9cea7e93054d4498b57f0bfe4b312d3f P ae62d02eff7749979f0c98a8907c918a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b229b3bf9b84796bb23853b0600c783" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42595 } } peers { permanent_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34589 } } peers { permanent_uuid: "ae62d02eff7749979f0c98a8907c918a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38807 } }
I20260928 20:19:33.335711 5386 ts_tablet_manager.cc:1428] T 9cea7e93054d4498b57f0bfe4b312d3f P ba5667d185cb4f0e8d923ae42f7105bf: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:33.335779 5384 ts_tablet_manager.cc:1428] T 9cea7e93054d4498b57f0bfe4b312d3f P ae62d02eff7749979f0c98a8907c918a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:33.335906 4864 catalog_manager.cc:4841] TS ba5667d185cb4f0e8d923ae42f7105bf (127.3.18.2:34589): tablet 6869be9d2b8a48b8940932ce2158a165 (table test-table [id=1209d70dcf9b479091536365d16bcfed]) successfully deleted
I20260928 20:19:33.336519 5385 tablet_bootstrap.cc:492] T 9cea7e93054d4498b57f0bfe4b312d3f P 2b229b3bf9b84796bb23853b0600c783: Bootstrap starting.
I20260928 20:19:33.336906 5385 tablet_bootstrap.cc:654] T 9cea7e93054d4498b57f0bfe4b312d3f P 2b229b3bf9b84796bb23853b0600c783: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:33.337642 5385 tablet_bootstrap.cc:492] T 9cea7e93054d4498b57f0bfe4b312d3f P 2b229b3bf9b84796bb23853b0600c783: No bootstrap required, opened a new log
I20260928 20:19:33.337718 5385 ts_tablet_manager.cc:1397] T 9cea7e93054d4498b57f0bfe4b312d3f P 2b229b3bf9b84796bb23853b0600c783: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 20:19:33.337900 5385 raft_consensus.cc:348] T 9cea7e93054d4498b57f0bfe4b312d3f P 2b229b3bf9b84796bb23853b0600c783 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b229b3bf9b84796bb23853b0600c783" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42595 } } peers { permanent_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34589 } } peers { permanent_uuid: "ae62d02eff7749979f0c98a8907c918a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38807 } }
I20260928 20:19:33.337990 5385 raft_consensus.cc:374] T 9cea7e93054d4498b57f0bfe4b312d3f P 2b229b3bf9b84796bb23853b0600c783 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:33.338019 5385 raft_consensus.cc:729] T 9cea7e93054d4498b57f0bfe4b312d3f P 2b229b3bf9b84796bb23853b0600c783 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b229b3bf9b84796bb23853b0600c783, State: Initialized, Role: FOLLOWER
I20260928 20:19:33.338064 5385 consensus_queue.cc:260] T 9cea7e93054d4498b57f0bfe4b312d3f P 2b229b3bf9b84796bb23853b0600c783 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b229b3bf9b84796bb23853b0600c783" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42595 } } peers { permanent_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34589 } } peers { permanent_uuid: "ae62d02eff7749979f0c98a8907c918a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38807 } }
I20260928 20:19:33.338169 5385 ts_tablet_manager.cc:1428] T 9cea7e93054d4498b57f0bfe4b312d3f P 2b229b3bf9b84796bb23853b0600c783: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 20:19:33.338860 5429 ts_tablet_manager.cc:1918] T 6869be9d2b8a48b8940932ce2158a165 P 2b229b3bf9b84796bb23853b0600c783: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 20:19:33.338944 5429 log.cc:1196] T 6869be9d2b8a48b8940932ce2158a165 P 2b229b3bf9b84796bb23853b0600c783: Deleting WAL directory at /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/ts-0/wal/wals/6869be9d2b8a48b8940932ce2158a165
I20260928 20:19:33.339213 5429 ts_tablet_manager.cc:1939] T 6869be9d2b8a48b8940932ce2158a165 P 2b229b3bf9b84796bb23853b0600c783: Deleting consensus metadata
I20260928 20:19:33.339744 4865 catalog_manager.cc:4841] TS 2b229b3bf9b84796bb23853b0600c783 (127.3.18.1:42595): tablet 6869be9d2b8a48b8940932ce2158a165 (table test-table [id=1209d70dcf9b479091536365d16bcfed]) successfully deleted
I20260928 20:19:33.390560 5392 raft_consensus.cc:482] T 9cea7e93054d4498b57f0bfe4b312d3f P 2b229b3bf9b84796bb23853b0600c783 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:19:33.390682 5392 raft_consensus.cc:504] T 9cea7e93054d4498b57f0bfe4b312d3f P 2b229b3bf9b84796bb23853b0600c783 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b229b3bf9b84796bb23853b0600c783" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42595 } } peers { permanent_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34589 } } peers { permanent_uuid: "ae62d02eff7749979f0c98a8907c918a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38807 } }
I20260928 20:19:33.391054 5392 leader_election.cc:288] T 9cea7e93054d4498b57f0bfe4b312d3f P 2b229b3bf9b84796bb23853b0600c783 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba5667d185cb4f0e8d923ae42f7105bf (127.3.18.2:34589), ae62d02eff7749979f0c98a8907c918a (127.3.18.3:38807)
I20260928 20:19:33.394619 5382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:33.391473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:48625 (local address 127.3.18.2:34589)
0928 20:19:33.391512 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:33.391514 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:33.391528 (+ 14us) server_negotiation.cc:407] Connection header received
0928 20:19:33.391658 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:33.391660 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:33.391671 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:33.391709 (+ 38us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:19:33.391951 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:33.392673 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:33.393825 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:33.394214 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:33.394302 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:33.394315 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:33.394502 (+ 187us) server_negotiation.cc:299] Negotiation successful
0928 20:19:33.394546 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 20:19:33.394639 5434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:33.391259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.2:34589 (local address 127.3.18.1:48625)
0928 20:19:33.391470 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:33.391482 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:33.391548 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:33.391820 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:33.391821 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:33.391822 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:19:33.391878 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:33.391879 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:33.392770 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:33.392771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:33.393719 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:33.393723 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:33.394320 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:33.394321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:33.394402 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:33.394406 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:19:33.394485 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 20:19:33.394540 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":2,"thread_start_us":103,"threads_started":1}
I20260928 20:19:33.395040 5175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cea7e93054d4498b57f0bfe4b312d3f" candidate_uuid: "2b229b3bf9b84796bb23853b0600c783" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" is_pre_election: true
I20260928 20:19:33.395174 5175 raft_consensus.cc:2454] T 9cea7e93054d4498b57f0bfe4b312d3f P ba5667d185cb4f0e8d923ae42f7105bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b229b3bf9b84796bb23853b0600c783 in term 0.
I20260928 20:19:33.395264 5409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:33.391329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:33011 (local address 127.3.18.3:38807)
0928 20:19:33.391397 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:33.391399 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:33.391602 (+ 203us) server_negotiation.cc:407] Connection header received
0928 20:19:33.391664 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:33.391666 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:33.391675 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:33.391739 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:19:33.392371 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:33.392977 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:33.394322 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:33.394864 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:33.394908 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:33.394917 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:33.395134 (+ 217us) server_negotiation.cc:299] Negotiation successful
0928 20:19:33.395189 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 20:19:33.395257 5433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:33.391258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.3:38807 (local address 127.3.18.1:33011)
0928 20:19:33.391470 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:33.391482 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:33.391593 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:33.391757 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:33.391758 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:33.391785 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:19:33.391872 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:33.391874 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:33.393094 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:33.393095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:33.394214 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:33.394217 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:33.394977 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:33.394977 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:33.395047 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:33.395051 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:19:33.395114 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 20:19:33.395152 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":130,"threads_started":1}
I20260928 20:19:33.395377 4968 leader_election.cc:302] T 9cea7e93054d4498b57f0bfe4b312d3f P 2b229b3bf9b84796bb23853b0600c783 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b229b3bf9b84796bb23853b0600c783, ba5667d185cb4f0e8d923ae42f7105bf; no voters:
I20260928 20:19:33.395530 5392 raft_consensus.cc:2781] T 9cea7e93054d4498b57f0bfe4b312d3f P 2b229b3bf9b84796bb23853b0600c783 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 20:19:33.395596 5392 raft_consensus.cc:482] T 9cea7e93054d4498b57f0bfe4b312d3f P 2b229b3bf9b84796bb23853b0600c783 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 20:19:33.395629 5392 raft_consensus.cc:3037] T 9cea7e93054d4498b57f0bfe4b312d3f P 2b229b3bf9b84796bb23853b0600c783 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:33.395560 5315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cea7e93054d4498b57f0bfe4b312d3f" candidate_uuid: "2b229b3bf9b84796bb23853b0600c783" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae62d02eff7749979f0c98a8907c918a" is_pre_election: true
I20260928 20:19:33.395697 5315 raft_consensus.cc:2454] T 9cea7e93054d4498b57f0bfe4b312d3f P ae62d02eff7749979f0c98a8907c918a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b229b3bf9b84796bb23853b0600c783 in term 0.
I20260928 20:19:33.396479 5392 raft_consensus.cc:504] T 9cea7e93054d4498b57f0bfe4b312d3f P 2b229b3bf9b84796bb23853b0600c783 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b229b3bf9b84796bb23853b0600c783" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42595 } } peers { permanent_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34589 } } peers { permanent_uuid: "ae62d02eff7749979f0c98a8907c918a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38807 } }
I20260928 20:19:33.396658 5392 leader_election.cc:288] T 9cea7e93054d4498b57f0bfe4b312d3f P 2b229b3bf9b84796bb23853b0600c783 [CANDIDATE]: Term 1 election: Requested vote from peers ba5667d185cb4f0e8d923ae42f7105bf (127.3.18.2:34589), ae62d02eff7749979f0c98a8907c918a (127.3.18.3:38807)
I20260928 20:19:33.396801 5175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cea7e93054d4498b57f0bfe4b312d3f" candidate_uuid: "2b229b3bf9b84796bb23853b0600c783" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba5667d185cb4f0e8d923ae42f7105bf"
I20260928 20:19:33.396849 5315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cea7e93054d4498b57f0bfe4b312d3f" candidate_uuid: "2b229b3bf9b84796bb23853b0600c783" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae62d02eff7749979f0c98a8907c918a"
I20260928 20:19:33.396888 5175 raft_consensus.cc:3037] T 9cea7e93054d4498b57f0bfe4b312d3f P ba5667d185cb4f0e8d923ae42f7105bf [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:33.396917 5315 raft_consensus.cc:3037] T 9cea7e93054d4498b57f0bfe4b312d3f P ae62d02eff7749979f0c98a8907c918a [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:33.397719 5175 raft_consensus.cc:2454] T 9cea7e93054d4498b57f0bfe4b312d3f P ba5667d185cb4f0e8d923ae42f7105bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b229b3bf9b84796bb23853b0600c783 in term 1.
I20260928 20:19:33.397801 5315 raft_consensus.cc:2454] T 9cea7e93054d4498b57f0bfe4b312d3f P ae62d02eff7749979f0c98a8907c918a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b229b3bf9b84796bb23853b0600c783 in term 1.
I20260928 20:19:33.397921 4968 leader_election.cc:302] T 9cea7e93054d4498b57f0bfe4b312d3f P 2b229b3bf9b84796bb23853b0600c783 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b229b3bf9b84796bb23853b0600c783, ba5667d185cb4f0e8d923ae42f7105bf; no voters:
I20260928 20:19:33.398018 5392 raft_consensus.cc:2781] T 9cea7e93054d4498b57f0bfe4b312d3f P 2b229b3bf9b84796bb23853b0600c783 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:19:33.398211 5392 raft_consensus.cc:686] T 9cea7e93054d4498b57f0bfe4b312d3f P 2b229b3bf9b84796bb23853b0600c783 [term 1 LEADER]: Becoming Leader. State: Replica: 2b229b3bf9b84796bb23853b0600c783, State: Running, Role: LEADER
I20260928 20:19:33.398296 5392 consensus_queue.cc:237] T 9cea7e93054d4498b57f0bfe4b312d3f P 2b229b3bf9b84796bb23853b0600c783 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b229b3bf9b84796bb23853b0600c783" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42595 } } peers { permanent_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34589 } } peers { permanent_uuid: "ae62d02eff7749979f0c98a8907c918a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38807 } }
I20260928 20:19:33.399135 4877 catalog_manager.cc:5495] T 9cea7e93054d4498b57f0bfe4b312d3f P 2b229b3bf9b84796bb23853b0600c783 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b229b3bf9b84796bb23853b0600c783 (127.3.18.1). New cstate: current_term: 1 leader_uuid: "2b229b3bf9b84796bb23853b0600c783" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b229b3bf9b84796bb23853b0600c783" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae62d02eff7749979f0c98a8907c918a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38807 } health_report { overall_health: UNKNOWN } } }
I20260928 20:19:33.413954 5381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:33.411221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42424 (local address 127.3.18.1:42595)
0928 20:19:33.411275 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:33.411277 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:33.411286 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:19:33.411325 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:33.411326 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:33.411349 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:33.411437 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:19:33.411532 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:33.412329 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:33.413094 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:33.413396 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:33.413470 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:33.413583 (+ 113us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:19:33.413726 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:19:33.413802 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:33.413843 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 20:19:33.413869 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 20:19:33.418991 5409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:33.416563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:60971 (local address 127.3.18.3:38807)
0928 20:19:33.416616 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:33.416617 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:33.416779 (+ 162us) server_negotiation.cc:407] Connection header received
0928 20:19:33.416854 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:33.416855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:33.416868 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:33.416938 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:19:33.417074 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:33.417568 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:33.418304 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:33.418489 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:33.418554 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:33.418736 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:19:33.418770 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:19:33.418828 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:33.418888 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 20:19:33.418911 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 20:19:33.419019 5439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:33.416498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.3:38807 (local address 127.3.18.1:60971)
0928 20:19:33.416714 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:33.416723 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:33.416793 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:33.416942 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:33.416943 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:33.416945 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 20:19:33.417007 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:33.417008 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:33.417652 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:33.417653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:33.418222 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:33.418224 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:33.418583 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:33.418584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:33.418648 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:33.418662 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 20:19:33.418828 (+ 166us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 20:19:33.418831 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 20:19:33.418884 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 20:19:33.418926 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":94,"threads_started":1}
I20260928 20:19:33.421170 5175 raft_consensus.cc:1261] T 9cea7e93054d4498b57f0bfe4b312d3f P ba5667d185cb4f0e8d923ae42f7105bf [term 1 FOLLOWER]: Refusing update from remote peer 2b229b3bf9b84796bb23853b0600c783: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:19:33.421255 5315 raft_consensus.cc:1261] T 9cea7e93054d4498b57f0bfe4b312d3f P ae62d02eff7749979f0c98a8907c918a [term 1 FOLLOWER]: Refusing update from remote peer 2b229b3bf9b84796bb23853b0600c783: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:19:33.421471 5435 consensus_queue.cc:1035] T 9cea7e93054d4498b57f0bfe4b312d3f P 2b229b3bf9b84796bb23853b0600c783 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:19:33.421634 5392 consensus_queue.cc:1035] T 9cea7e93054d4498b57f0bfe4b312d3f P 2b229b3bf9b84796bb23853b0600c783 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae62d02eff7749979f0c98a8907c918a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:19:33.429072 5381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:33.426608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:33313 (local address 127.3.18.1:42595)
0928 20:19:33.426645 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:33.426648 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:33.426677 (+ 29us) server_negotiation.cc:407] Connection header received
0928 20:19:33.426742 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:33.426743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:33.426765 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:33.426813 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:19:33.426949 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:33.427500 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:33.428327 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:33.428598 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:33.428686 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:33.428829 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:19:33.428863 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:19:33.428916 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:33.428978 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 20:19:33.429000 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 20:19:33.429097 5422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:33.426574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.1:42595 (local address 127.3.18.3:33313)
0928 20:19:33.426625 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:33.426647 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:33.426691 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:33.426853 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:33.426854 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:33.426855 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 20:19:33.426896 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:33.426898 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:33.427582 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:33.427583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:33.428214 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:33.428217 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:33.428688 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:33.428689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:33.428754 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:33.428761 (+ 7us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 20:19:33.428925 (+ 164us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 20:19:33.428926 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 20:19:33.428975 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 20:19:33.429016 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 20:19:33.429494 5411 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 20:19:34.954447 4877 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:19:34.954551 4877 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:19:34.955641 4877 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cddefad8370040909095f66cc0817b0e: Sending DeleteTablet for 3 replicas of tablet 9cea7e93054d4498b57f0bfe4b312d3f
I20260928 20:19:34.955960 5294 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cea7e93054d4498b57f0bfe4b312d3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:19:34 UTC) from {username='kudu', principal='kudu/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:48108
I20260928 20:19:34.956030 5015 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cea7e93054d4498b57f0bfe4b312d3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:19:34 UTC) from {username='kudu', principal='kudu/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:42410
I20260928 20:19:34.958129 5155 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cea7e93054d4498b57f0bfe4b312d3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:19:34 UTC) from {username='kudu', principal='kudu/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:49984
I20260928 20:19:34.958375 5467 tablet_replica.cc:330] T 9cea7e93054d4498b57f0bfe4b312d3f P ba5667d185cb4f0e8d923ae42f7105bf: stopping tablet replica
I20260928 20:19:34.958493 5467 raft_consensus.cc:2229] T 9cea7e93054d4498b57f0bfe4b312d3f P ba5667d185cb4f0e8d923ae42f7105bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:19:34.958566 5467 raft_consensus.cc:2258] T 9cea7e93054d4498b57f0bfe4b312d3f P ba5667d185cb4f0e8d923ae42f7105bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:19:34.958650 5468 tablet_replica.cc:330] T 9cea7e93054d4498b57f0bfe4b312d3f P ae62d02eff7749979f0c98a8907c918a: stopping tablet replica
I20260928 20:19:34.958751 5468 raft_consensus.cc:2229] T 9cea7e93054d4498b57f0bfe4b312d3f P ae62d02eff7749979f0c98a8907c918a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:19:34.958804 5468 raft_consensus.cc:2258] T 9cea7e93054d4498b57f0bfe4b312d3f P ae62d02eff7749979f0c98a8907c918a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:19:34.958916 5467 ts_tablet_manager.cc:1905] T 9cea7e93054d4498b57f0bfe4b312d3f P ba5667d185cb4f0e8d923ae42f7105bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:19:34.959067 5468 ts_tablet_manager.cc:1905] T 9cea7e93054d4498b57f0bfe4b312d3f P ae62d02eff7749979f0c98a8907c918a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:19:34.961018 5465 tablet_replica.cc:330] T 9cea7e93054d4498b57f0bfe4b312d3f P 2b229b3bf9b84796bb23853b0600c783: stopping tablet replica
I20260928 20:19:34.961094 5465 raft_consensus.cc:2229] T 9cea7e93054d4498b57f0bfe4b312d3f P 2b229b3bf9b84796bb23853b0600c783 [term 1 LEADER]: Raft consensus shutting down.
I20260928 20:19:34.961200 5465 raft_consensus.cc:2258] T 9cea7e93054d4498b57f0bfe4b312d3f P 2b229b3bf9b84796bb23853b0600c783 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:19:34.961252 5467 ts_tablet_manager.cc:1918] T 9cea7e93054d4498b57f0bfe4b312d3f P ba5667d185cb4f0e8d923ae42f7105bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 20:19:34.961315 5467 log.cc:1196] T 9cea7e93054d4498b57f0bfe4b312d3f P ba5667d185cb4f0e8d923ae42f7105bf: Deleting WAL directory at /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/ts-1/wal/wals/9cea7e93054d4498b57f0bfe4b312d3f
I20260928 20:19:34.961340 5468 ts_tablet_manager.cc:1918] T 9cea7e93054d4498b57f0bfe4b312d3f P ae62d02eff7749979f0c98a8907c918a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 20:19:34.961398 5468 log.cc:1196] T 9cea7e93054d4498b57f0bfe4b312d3f P ae62d02eff7749979f0c98a8907c918a: Deleting WAL directory at /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/ts-2/wal/wals/9cea7e93054d4498b57f0bfe4b312d3f
I20260928 20:19:34.961588 5467 ts_tablet_manager.cc:1939] T 9cea7e93054d4498b57f0bfe4b312d3f P ba5667d185cb4f0e8d923ae42f7105bf: Deleting consensus metadata
I20260928 20:19:34.961632 5468 ts_tablet_manager.cc:1939] T 9cea7e93054d4498b57f0bfe4b312d3f P ae62d02eff7749979f0c98a8907c918a: Deleting consensus metadata
I20260928 20:19:34.961978 4864 catalog_manager.cc:4841] TS ba5667d185cb4f0e8d923ae42f7105bf (127.3.18.2:34589): tablet 9cea7e93054d4498b57f0bfe4b312d3f (table test-table [id=fe51b6f3788a4006a137ad5696559a3f]) successfully deleted
I20260928 20:19:34.962074 4863 catalog_manager.cc:4841] TS ae62d02eff7749979f0c98a8907c918a (127.3.18.3:38807): tablet 9cea7e93054d4498b57f0bfe4b312d3f (table test-table [id=fe51b6f3788a4006a137ad5696559a3f]) successfully deleted
I20260928 20:19:34.962183 5465 ts_tablet_manager.cc:1905] T 9cea7e93054d4498b57f0bfe4b312d3f P 2b229b3bf9b84796bb23853b0600c783: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:19:34.963939 5465 ts_tablet_manager.cc:1918] T 9cea7e93054d4498b57f0bfe4b312d3f P 2b229b3bf9b84796bb23853b0600c783: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 20:19:34.963991 5465 log.cc:1196] T 9cea7e93054d4498b57f0bfe4b312d3f P 2b229b3bf9b84796bb23853b0600c783: Deleting WAL directory at /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/ts-0/wal/wals/9cea7e93054d4498b57f0bfe4b312d3f
I20260928 20:19:34.964224 5465 ts_tablet_manager.cc:1939] T 9cea7e93054d4498b57f0bfe4b312d3f P 2b229b3bf9b84796bb23853b0600c783: Deleting consensus metadata
I20260928 20:19:34.964738 4865 catalog_manager.cc:4841] TS 2b229b3bf9b84796bb23853b0600c783 (127.3.18.1:42595): tablet 9cea7e93054d4498b57f0bfe4b312d3f (table test-table [id=fe51b6f3788a4006a137ad5696559a3f]) successfully deleted
Sep 28 20:19:34 dist-test-slave-b83v krb5kdc[4833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626774, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 20:19:34 dist-test-slave-b83v krb5kdc[4833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626774, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:19:34.984493 5475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:34.975953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60448 (local address 127.3.18.62:39065)
0928 20:19:34.976122 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:34.976124 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:34.976136 (+ 12us) server_negotiation.cc:407] Connection header received
0928 20:19:34.976282 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:34.976283 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:34.976296 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:34.976364 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:34.977066 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:34.977581 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:34.978361 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:34.978555 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:34.978613 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:34.981309 (+ 2696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:34.981311 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:34.981311 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:34.981319 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:34.982941 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:34.983395 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:34.983397 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:34.983397 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:34.983462 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:34.983747 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:34.983748 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:34.983749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:34.983964 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:34.984073 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:34.984302 (+ 229us) server_negotiation.cc:299] Negotiation successful
0928 20:19:34.984401 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":116,"threads_started":1}
I20260928 20:19:34.986552 4877 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60448:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 20:19:34.986760 4877 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 20:19:34.988338 5015 tablet_service.cc:1467] Processing CreateTablet for tablet 2232733576264f2781be093fba857313 (DEFAULT_TABLE table=test-table [id=b4886ea4df274124b8030e8cac657868]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:19:34.988438 5294 tablet_service.cc:1467] Processing CreateTablet for tablet 2232733576264f2781be093fba857313 (DEFAULT_TABLE table=test-table [id=b4886ea4df274124b8030e8cac657868]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:19:34.988519 5015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2232733576264f2781be093fba857313. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:34.988544 5294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2232733576264f2781be093fba857313. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:34.989964 5476 tablet_bootstrap.cc:492] T 2232733576264f2781be093fba857313 P 2b229b3bf9b84796bb23853b0600c783: Bootstrap starting.
I20260928 20:19:34.990075 5155 tablet_service.cc:1467] Processing CreateTablet for tablet 2232733576264f2781be093fba857313 (DEFAULT_TABLE table=test-table [id=b4886ea4df274124b8030e8cac657868]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:19:34.990202 5155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2232733576264f2781be093fba857313. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:34.990443 5476 tablet_bootstrap.cc:654] T 2232733576264f2781be093fba857313 P 2b229b3bf9b84796bb23853b0600c783: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:34.991057 5476 tablet_bootstrap.cc:492] T 2232733576264f2781be093fba857313 P 2b229b3bf9b84796bb23853b0600c783: No bootstrap required, opened a new log
I20260928 20:19:34.991129 5476 ts_tablet_manager.cc:1397] T 2232733576264f2781be093fba857313 P 2b229b3bf9b84796bb23853b0600c783: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 20:19:34.991354 5476 raft_consensus.cc:348] T 2232733576264f2781be093fba857313 P 2b229b3bf9b84796bb23853b0600c783 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b229b3bf9b84796bb23853b0600c783" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42595 } } peers { permanent_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34589 } } peers { permanent_uuid: "ae62d02eff7749979f0c98a8907c918a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38807 } }
I20260928 20:19:34.991446 5476 raft_consensus.cc:374] T 2232733576264f2781be093fba857313 P 2b229b3bf9b84796bb23853b0600c783 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:34.991482 5476 raft_consensus.cc:729] T 2232733576264f2781be093fba857313 P 2b229b3bf9b84796bb23853b0600c783 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b229b3bf9b84796bb23853b0600c783, State: Initialized, Role: FOLLOWER
I20260928 20:19:34.991534 5476 consensus_queue.cc:260] T 2232733576264f2781be093fba857313 P 2b229b3bf9b84796bb23853b0600c783 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b229b3bf9b84796bb23853b0600c783" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42595 } } peers { permanent_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34589 } } peers { permanent_uuid: "ae62d02eff7749979f0c98a8907c918a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38807 } }
I20260928 20:19:34.991590 5477 tablet_bootstrap.cc:492] T 2232733576264f2781be093fba857313 P ae62d02eff7749979f0c98a8907c918a: Bootstrap starting.
I20260928 20:19:34.991797 5476 ts_tablet_manager.cc:1428] T 2232733576264f2781be093fba857313 P 2b229b3bf9b84796bb23853b0600c783: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 20:19:34.991709 5479 tablet_bootstrap.cc:492] T 2232733576264f2781be093fba857313 P ba5667d185cb4f0e8d923ae42f7105bf: Bootstrap starting.
I20260928 20:19:34.991991 5477 tablet_bootstrap.cc:654] T 2232733576264f2781be093fba857313 P ae62d02eff7749979f0c98a8907c918a: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:34.992408 5479 tablet_bootstrap.cc:654] T 2232733576264f2781be093fba857313 P ba5667d185cb4f0e8d923ae42f7105bf: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:34.992585 5477 tablet_bootstrap.cc:492] T 2232733576264f2781be093fba857313 P ae62d02eff7749979f0c98a8907c918a: No bootstrap required, opened a new log
I20260928 20:19:34.992659 5477 ts_tablet_manager.cc:1397] T 2232733576264f2781be093fba857313 P ae62d02eff7749979f0c98a8907c918a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 20:19:34.992794 5477 raft_consensus.cc:348] T 2232733576264f2781be093fba857313 P ae62d02eff7749979f0c98a8907c918a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b229b3bf9b84796bb23853b0600c783" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42595 } } peers { permanent_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34589 } } peers { permanent_uuid: "ae62d02eff7749979f0c98a8907c918a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38807 } }
I20260928 20:19:34.992877 5477 raft_consensus.cc:374] T 2232733576264f2781be093fba857313 P ae62d02eff7749979f0c98a8907c918a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:34.992909 5477 raft_consensus.cc:729] T 2232733576264f2781be093fba857313 P ae62d02eff7749979f0c98a8907c918a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae62d02eff7749979f0c98a8907c918a, State: Initialized, Role: FOLLOWER
I20260928 20:19:34.992957 5477 consensus_queue.cc:260] T 2232733576264f2781be093fba857313 P ae62d02eff7749979f0c98a8907c918a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b229b3bf9b84796bb23853b0600c783" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42595 } } peers { permanent_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34589 } } peers { permanent_uuid: "ae62d02eff7749979f0c98a8907c918a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38807 } }
I20260928 20:19:34.993036 5479 tablet_bootstrap.cc:492] T 2232733576264f2781be093fba857313 P ba5667d185cb4f0e8d923ae42f7105bf: No bootstrap required, opened a new log
I20260928 20:19:34.993064 5477 ts_tablet_manager.cc:1428] T 2232733576264f2781be093fba857313 P ae62d02eff7749979f0c98a8907c918a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:34.993081 5479 ts_tablet_manager.cc:1397] T 2232733576264f2781be093fba857313 P ba5667d185cb4f0e8d923ae42f7105bf: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 20:19:34.993217 5479 raft_consensus.cc:348] T 2232733576264f2781be093fba857313 P ba5667d185cb4f0e8d923ae42f7105bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b229b3bf9b84796bb23853b0600c783" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42595 } } peers { permanent_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34589 } } peers { permanent_uuid: "ae62d02eff7749979f0c98a8907c918a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38807 } }
I20260928 20:19:34.993287 5479 raft_consensus.cc:374] T 2232733576264f2781be093fba857313 P ba5667d185cb4f0e8d923ae42f7105bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:34.993314 5479 raft_consensus.cc:729] T 2232733576264f2781be093fba857313 P ba5667d185cb4f0e8d923ae42f7105bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba5667d185cb4f0e8d923ae42f7105bf, State: Initialized, Role: FOLLOWER
I20260928 20:19:34.993382 5479 consensus_queue.cc:260] T 2232733576264f2781be093fba857313 P ba5667d185cb4f0e8d923ae42f7105bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b229b3bf9b84796bb23853b0600c783" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42595 } } peers { permanent_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34589 } } peers { permanent_uuid: "ae62d02eff7749979f0c98a8907c918a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38807 } }
I20260928 20:19:34.993579 5479 ts_tablet_manager.cc:1428] T 2232733576264f2781be093fba857313 P ba5667d185cb4f0e8d923ae42f7105bf: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 20:19:35.122146 5435 raft_consensus.cc:482] T 2232733576264f2781be093fba857313 P 2b229b3bf9b84796bb23853b0600c783 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:19:35.122296 5435 raft_consensus.cc:504] T 2232733576264f2781be093fba857313 P 2b229b3bf9b84796bb23853b0600c783 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b229b3bf9b84796bb23853b0600c783" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42595 } } peers { permanent_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34589 } } peers { permanent_uuid: "ae62d02eff7749979f0c98a8907c918a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38807 } }
I20260928 20:19:35.122506 5435 leader_election.cc:288] T 2232733576264f2781be093fba857313 P 2b229b3bf9b84796bb23853b0600c783 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba5667d185cb4f0e8d923ae42f7105bf (127.3.18.2:34589), ae62d02eff7749979f0c98a8907c918a (127.3.18.3:38807)
I20260928 20:19:35.122731 5175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2232733576264f2781be093fba857313" candidate_uuid: "2b229b3bf9b84796bb23853b0600c783" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" is_pre_election: true
I20260928 20:19:35.122738 5315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2232733576264f2781be093fba857313" candidate_uuid: "2b229b3bf9b84796bb23853b0600c783" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae62d02eff7749979f0c98a8907c918a" is_pre_election: true
I20260928 20:19:35.122848 5175 raft_consensus.cc:2454] T 2232733576264f2781be093fba857313 P ba5667d185cb4f0e8d923ae42f7105bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b229b3bf9b84796bb23853b0600c783 in term 0.
I20260928 20:19:35.122876 5315 raft_consensus.cc:2454] T 2232733576264f2781be093fba857313 P ae62d02eff7749979f0c98a8907c918a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b229b3bf9b84796bb23853b0600c783 in term 0.
I20260928 20:19:35.123030 4968 leader_election.cc:302] T 2232733576264f2781be093fba857313 P 2b229b3bf9b84796bb23853b0600c783 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b229b3bf9b84796bb23853b0600c783, ba5667d185cb4f0e8d923ae42f7105bf; no voters:
I20260928 20:19:35.123155 5435 raft_consensus.cc:2781] T 2232733576264f2781be093fba857313 P 2b229b3bf9b84796bb23853b0600c783 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 20:19:35.123220 5435 raft_consensus.cc:482] T 2232733576264f2781be093fba857313 P 2b229b3bf9b84796bb23853b0600c783 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 20:19:35.123252 5435 raft_consensus.cc:3037] T 2232733576264f2781be093fba857313 P 2b229b3bf9b84796bb23853b0600c783 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:35.124035 5435 raft_consensus.cc:504] T 2232733576264f2781be093fba857313 P 2b229b3bf9b84796bb23853b0600c783 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b229b3bf9b84796bb23853b0600c783" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42595 } } peers { permanent_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34589 } } peers { permanent_uuid: "ae62d02eff7749979f0c98a8907c918a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38807 } }
I20260928 20:19:35.124193 5435 leader_election.cc:288] T 2232733576264f2781be093fba857313 P 2b229b3bf9b84796bb23853b0600c783 [CANDIDATE]: Term 1 election: Requested vote from peers ba5667d185cb4f0e8d923ae42f7105bf (127.3.18.2:34589), ae62d02eff7749979f0c98a8907c918a (127.3.18.3:38807)
I20260928 20:19:35.124384 5315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2232733576264f2781be093fba857313" candidate_uuid: "2b229b3bf9b84796bb23853b0600c783" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae62d02eff7749979f0c98a8907c918a"
I20260928 20:19:35.124467 5315 raft_consensus.cc:3037] T 2232733576264f2781be093fba857313 P ae62d02eff7749979f0c98a8907c918a [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:35.124428 5175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2232733576264f2781be093fba857313" candidate_uuid: "2b229b3bf9b84796bb23853b0600c783" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba5667d185cb4f0e8d923ae42f7105bf"
I20260928 20:19:35.124519 5175 raft_consensus.cc:3037] T 2232733576264f2781be093fba857313 P ba5667d185cb4f0e8d923ae42f7105bf [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:35.125104 5175 raft_consensus.cc:2454] T 2232733576264f2781be093fba857313 P ba5667d185cb4f0e8d923ae42f7105bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b229b3bf9b84796bb23853b0600c783 in term 1.
I20260928 20:19:35.125182 5315 raft_consensus.cc:2454] T 2232733576264f2781be093fba857313 P ae62d02eff7749979f0c98a8907c918a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b229b3bf9b84796bb23853b0600c783 in term 1.
I20260928 20:19:35.125345 4968 leader_election.cc:302] T 2232733576264f2781be093fba857313 P 2b229b3bf9b84796bb23853b0600c783 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b229b3bf9b84796bb23853b0600c783, ba5667d185cb4f0e8d923ae42f7105bf; no voters:
I20260928 20:19:35.125474 5435 raft_consensus.cc:2781] T 2232733576264f2781be093fba857313 P 2b229b3bf9b84796bb23853b0600c783 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:19:35.125535 5435 raft_consensus.cc:686] T 2232733576264f2781be093fba857313 P 2b229b3bf9b84796bb23853b0600c783 [term 1 LEADER]: Becoming Leader. State: Replica: 2b229b3bf9b84796bb23853b0600c783, State: Running, Role: LEADER
I20260928 20:19:35.125590 5435 consensus_queue.cc:237] T 2232733576264f2781be093fba857313 P 2b229b3bf9b84796bb23853b0600c783 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b229b3bf9b84796bb23853b0600c783" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42595 } } peers { permanent_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34589 } } peers { permanent_uuid: "ae62d02eff7749979f0c98a8907c918a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38807 } }
I20260928 20:19:35.126425 4877 catalog_manager.cc:5495] T 2232733576264f2781be093fba857313 P 2b229b3bf9b84796bb23853b0600c783 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b229b3bf9b84796bb23853b0600c783 (127.3.18.1). New cstate: current_term: 1 leader_uuid: "2b229b3bf9b84796bb23853b0600c783" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b229b3bf9b84796bb23853b0600c783" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae62d02eff7749979f0c98a8907c918a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38807 } health_report { overall_health: UNKNOWN } } }
I20260928 20:19:35.141716 5485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:35.138996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42440 (local address 127.3.18.1:42595)
0928 20:19:35.139161 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:35.139163 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:35.139169 (+ 6us) server_negotiation.cc:407] Connection header received
0928 20:19:35.139236 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:35.139237 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:35.139246 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:35.139301 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:19:35.139448 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:35.140136 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:35.141022 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:35.141255 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:35.141316 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:35.141501 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:19:35.141531 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:19:35.141567 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:35.141599 (+ 32us) server_negotiation.cc:299] Negotiation successful
0928 20:19:35.141618 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":101,"threads_started":1}
I20260928 20:19:35.144775 5315 raft_consensus.cc:1261] T 2232733576264f2781be093fba857313 P ae62d02eff7749979f0c98a8907c918a [term 1 FOLLOWER]: Refusing update from remote peer 2b229b3bf9b84796bb23853b0600c783: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:19:35.144810 5175 raft_consensus.cc:1261] T 2232733576264f2781be093fba857313 P ba5667d185cb4f0e8d923ae42f7105bf [term 1 FOLLOWER]: Refusing update from remote peer 2b229b3bf9b84796bb23853b0600c783: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:19:35.145207 5435 consensus_queue.cc:1035] T 2232733576264f2781be093fba857313 P 2b229b3bf9b84796bb23853b0600c783 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae62d02eff7749979f0c98a8907c918a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:19:35.145429 5456 consensus_queue.cc:1035] T 2232733576264f2781be093fba857313 P 2b229b3bf9b84796bb23853b0600c783 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:19:35.157685 4878 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60448:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:19:35.157815 4878 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60448:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:19:35.158747 4878 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cddefad8370040909095f66cc0817b0e: Sending DeleteTablet for 3 replicas of tablet 2232733576264f2781be093fba857313
I20260928 20:19:35.159058 5155 tablet_service.cc:1514] Processing DeleteTablet for tablet 2232733576264f2781be093fba857313 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:19:35 UTC) from {username='kudu', principal='kudu/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:49984
I20260928 20:19:35.159075 5015 tablet_service.cc:1514] Processing DeleteTablet for tablet 2232733576264f2781be093fba857313 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:19:35 UTC) from {username='kudu', principal='kudu/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:42410
I20260928 20:19:35.159235 5465 tablet_replica.cc:330] T 2232733576264f2781be093fba857313 P 2b229b3bf9b84796bb23853b0600c783: stopping tablet replica
I20260928 20:19:35.159236 5467 tablet_replica.cc:330] T 2232733576264f2781be093fba857313 P ba5667d185cb4f0e8d923ae42f7105bf: stopping tablet replica
I20260928 20:19:35.159302 5465 raft_consensus.cc:2229] T 2232733576264f2781be093fba857313 P 2b229b3bf9b84796bb23853b0600c783 [term 1 LEADER]: Raft consensus shutting down.
I20260928 20:19:35.159317 5467 raft_consensus.cc:2229] T 2232733576264f2781be093fba857313 P ba5667d185cb4f0e8d923ae42f7105bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:19:35.159374 5465 raft_consensus.cc:2258] T 2232733576264f2781be093fba857313 P 2b229b3bf9b84796bb23853b0600c783 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:19:35.159385 5467 raft_consensus.cc:2258] T 2232733576264f2781be093fba857313 P ba5667d185cb4f0e8d923ae42f7105bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:19:35.159746 5467 ts_tablet_manager.cc:1905] T 2232733576264f2781be093fba857313 P ba5667d185cb4f0e8d923ae42f7105bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:19:35.160364 5294 tablet_service.cc:1514] Processing DeleteTablet for tablet 2232733576264f2781be093fba857313 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:19:35 UTC) from {username='kudu', principal='kudu/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:48108
I20260928 20:19:35.160529 5468 tablet_replica.cc:330] T 2232733576264f2781be093fba857313 P ae62d02eff7749979f0c98a8907c918a: stopping tablet replica
I20260928 20:19:35.160604 5468 raft_consensus.cc:2229] T 2232733576264f2781be093fba857313 P ae62d02eff7749979f0c98a8907c918a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:19:35.160647 4878 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60448:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 20:19:35.160845 4878 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 20:19:35.160874 5468 raft_consensus.cc:2258] T 2232733576264f2781be093fba857313 P ae62d02eff7749979f0c98a8907c918a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:19:35.161496 5468 ts_tablet_manager.cc:1905] T 2232733576264f2781be093fba857313 P ae62d02eff7749979f0c98a8907c918a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:19:35.162161 5465 ts_tablet_manager.cc:1905] T 2232733576264f2781be093fba857313 P 2b229b3bf9b84796bb23853b0600c783: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:19:35.162323 5467 ts_tablet_manager.cc:1918] T 2232733576264f2781be093fba857313 P ba5667d185cb4f0e8d923ae42f7105bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 20:19:35.162392 5467 log.cc:1196] T 2232733576264f2781be093fba857313 P ba5667d185cb4f0e8d923ae42f7105bf: Deleting WAL directory at /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/ts-1/wal/wals/2232733576264f2781be093fba857313
I20260928 20:19:35.162657 5467 ts_tablet_manager.cc:1939] T 2232733576264f2781be093fba857313 P ba5667d185cb4f0e8d923ae42f7105bf: Deleting consensus metadata
I20260928 20:19:35.163136 5015 tablet_service.cc:1467] Processing CreateTablet for tablet b3b61c76bd9c443586b463de9464f811 (DEFAULT_TABLE table=test-table [id=949b88ee3c964c3caff085bdc374c21a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:19:35.163283 5015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3b61c76bd9c443586b463de9464f811. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:35.163394 5468 ts_tablet_manager.cc:1918] T 2232733576264f2781be093fba857313 P ae62d02eff7749979f0c98a8907c918a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 20:19:35.163447 5468 log.cc:1196] T 2232733576264f2781be093fba857313 P ae62d02eff7749979f0c98a8907c918a: Deleting WAL directory at /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/ts-2/wal/wals/2232733576264f2781be093fba857313
I20260928 20:19:35.163609 4864 catalog_manager.cc:4841] TS ba5667d185cb4f0e8d923ae42f7105bf (127.3.18.2:34589): tablet 2232733576264f2781be093fba857313 (table test-table [id=b4886ea4df274124b8030e8cac657868]) successfully deleted
I20260928 20:19:35.163663 5468 ts_tablet_manager.cc:1939] T 2232733576264f2781be093fba857313 P ae62d02eff7749979f0c98a8907c918a: Deleting consensus metadata
I20260928 20:19:35.163875 5155 tablet_service.cc:1467] Processing CreateTablet for tablet b3b61c76bd9c443586b463de9464f811 (DEFAULT_TABLE table=test-table [id=949b88ee3c964c3caff085bdc374c21a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:19:35.163978 5155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3b61c76bd9c443586b463de9464f811. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:35.164002 5465 ts_tablet_manager.cc:1918] T 2232733576264f2781be093fba857313 P 2b229b3bf9b84796bb23853b0600c783: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 20:19:35.164062 5465 log.cc:1196] T 2232733576264f2781be093fba857313 P 2b229b3bf9b84796bb23853b0600c783: Deleting WAL directory at /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/ts-0/wal/wals/2232733576264f2781be093fba857313
I20260928 20:19:35.164218 5465 ts_tablet_manager.cc:1939] T 2232733576264f2781be093fba857313 P 2b229b3bf9b84796bb23853b0600c783: Deleting consensus metadata
I20260928 20:19:35.164475 4863 catalog_manager.cc:4841] TS ae62d02eff7749979f0c98a8907c918a (127.3.18.3:38807): tablet 2232733576264f2781be093fba857313 (table test-table [id=b4886ea4df274124b8030e8cac657868]) successfully deleted
I20260928 20:19:35.164558 5294 tablet_service.cc:1467] Processing CreateTablet for tablet b3b61c76bd9c443586b463de9464f811 (DEFAULT_TABLE table=test-table [id=949b88ee3c964c3caff085bdc374c21a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:19:35.164606 5476 tablet_bootstrap.cc:492] T b3b61c76bd9c443586b463de9464f811 P 2b229b3bf9b84796bb23853b0600c783: Bootstrap starting.
I20260928 20:19:35.164654 5294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3b61c76bd9c443586b463de9464f811. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:35.164927 5476 tablet_bootstrap.cc:654] T b3b61c76bd9c443586b463de9464f811 P 2b229b3bf9b84796bb23853b0600c783: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:35.165585 5476 tablet_bootstrap.cc:492] T b3b61c76bd9c443586b463de9464f811 P 2b229b3bf9b84796bb23853b0600c783: No bootstrap required, opened a new log
I20260928 20:19:35.165647 5476 ts_tablet_manager.cc:1397] T b3b61c76bd9c443586b463de9464f811 P 2b229b3bf9b84796bb23853b0600c783: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 20:19:35.165777 5476 raft_consensus.cc:348] T b3b61c76bd9c443586b463de9464f811 P 2b229b3bf9b84796bb23853b0600c783 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b229b3bf9b84796bb23853b0600c783" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42595 } } peers { permanent_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34589 } } peers { permanent_uuid: "ae62d02eff7749979f0c98a8907c918a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38807 } }
I20260928 20:19:35.165851 5476 raft_consensus.cc:374] T b3b61c76bd9c443586b463de9464f811 P 2b229b3bf9b84796bb23853b0600c783 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:35.165871 5476 raft_consensus.cc:729] T b3b61c76bd9c443586b463de9464f811 P 2b229b3bf9b84796bb23853b0600c783 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b229b3bf9b84796bb23853b0600c783, State: Initialized, Role: FOLLOWER
I20260928 20:19:35.165922 5476 consensus_queue.cc:260] T b3b61c76bd9c443586b463de9464f811 P 2b229b3bf9b84796bb23853b0600c783 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b229b3bf9b84796bb23853b0600c783" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42595 } } peers { permanent_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34589 } } peers { permanent_uuid: "ae62d02eff7749979f0c98a8907c918a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38807 } }
I20260928 20:19:35.166021 5476 ts_tablet_manager.cc:1428] T b3b61c76bd9c443586b463de9464f811 P 2b229b3bf9b84796bb23853b0600c783: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:35.166030 5479 tablet_bootstrap.cc:492] T b3b61c76bd9c443586b463de9464f811 P ba5667d185cb4f0e8d923ae42f7105bf: Bootstrap starting.
I20260928 20:19:35.166121 4865 catalog_manager.cc:4841] TS 2b229b3bf9b84796bb23853b0600c783 (127.3.18.1:42595): tablet 2232733576264f2781be093fba857313 (table test-table [id=b4886ea4df274124b8030e8cac657868]) successfully deleted
I20260928 20:19:35.166443 5479 tablet_bootstrap.cc:654] T b3b61c76bd9c443586b463de9464f811 P ba5667d185cb4f0e8d923ae42f7105bf: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:35.167002 5477 tablet_bootstrap.cc:492] T b3b61c76bd9c443586b463de9464f811 P ae62d02eff7749979f0c98a8907c918a: Bootstrap starting.
I20260928 20:19:35.167291 5477 tablet_bootstrap.cc:654] T b3b61c76bd9c443586b463de9464f811 P ae62d02eff7749979f0c98a8907c918a: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:35.167557 5479 tablet_bootstrap.cc:492] T b3b61c76bd9c443586b463de9464f811 P ba5667d185cb4f0e8d923ae42f7105bf: No bootstrap required, opened a new log
I20260928 20:19:35.167606 5479 ts_tablet_manager.cc:1397] T b3b61c76bd9c443586b463de9464f811 P ba5667d185cb4f0e8d923ae42f7105bf: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 20:19:35.167802 5479 raft_consensus.cc:348] T b3b61c76bd9c443586b463de9464f811 P ba5667d185cb4f0e8d923ae42f7105bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b229b3bf9b84796bb23853b0600c783" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42595 } } peers { permanent_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34589 } } peers { permanent_uuid: "ae62d02eff7749979f0c98a8907c918a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38807 } }
I20260928 20:19:35.167899 5479 raft_consensus.cc:374] T b3b61c76bd9c443586b463de9464f811 P ba5667d185cb4f0e8d923ae42f7105bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:35.167918 5479 raft_consensus.cc:729] T b3b61c76bd9c443586b463de9464f811 P ba5667d185cb4f0e8d923ae42f7105bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba5667d185cb4f0e8d923ae42f7105bf, State: Initialized, Role: FOLLOWER
I20260928 20:19:35.167956 5479 consensus_queue.cc:260] T b3b61c76bd9c443586b463de9464f811 P ba5667d185cb4f0e8d923ae42f7105bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b229b3bf9b84796bb23853b0600c783" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42595 } } peers { permanent_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34589 } } peers { permanent_uuid: "ae62d02eff7749979f0c98a8907c918a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38807 } }
I20260928 20:19:35.168041 5479 ts_tablet_manager.cc:1428] T b3b61c76bd9c443586b463de9464f811 P ba5667d185cb4f0e8d923ae42f7105bf: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 20:19:35.168308 5477 tablet_bootstrap.cc:492] T b3b61c76bd9c443586b463de9464f811 P ae62d02eff7749979f0c98a8907c918a: No bootstrap required, opened a new log
I20260928 20:19:35.168341 5477 ts_tablet_manager.cc:1397] T b3b61c76bd9c443586b463de9464f811 P ae62d02eff7749979f0c98a8907c918a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 20:19:35.168424 5477 raft_consensus.cc:348] T b3b61c76bd9c443586b463de9464f811 P ae62d02eff7749979f0c98a8907c918a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b229b3bf9b84796bb23853b0600c783" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42595 } } peers { permanent_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34589 } } peers { permanent_uuid: "ae62d02eff7749979f0c98a8907c918a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38807 } }
I20260928 20:19:35.168469 5477 raft_consensus.cc:374] T b3b61c76bd9c443586b463de9464f811 P ae62d02eff7749979f0c98a8907c918a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:35.168481 5477 raft_consensus.cc:729] T b3b61c76bd9c443586b463de9464f811 P ae62d02eff7749979f0c98a8907c918a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae62d02eff7749979f0c98a8907c918a, State: Initialized, Role: FOLLOWER
I20260928 20:19:35.168516 5477 consensus_queue.cc:260] T b3b61c76bd9c443586b463de9464f811 P ae62d02eff7749979f0c98a8907c918a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b229b3bf9b84796bb23853b0600c783" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42595 } } peers { permanent_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34589 } } peers { permanent_uuid: "ae62d02eff7749979f0c98a8907c918a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38807 } }
I20260928 20:19:35.168581 5477 ts_tablet_manager.cc:1428] T b3b61c76bd9c443586b463de9464f811 P ae62d02eff7749979f0c98a8907c918a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:35.263962 5482 raft_consensus.cc:482] T b3b61c76bd9c443586b463de9464f811 P ba5667d185cb4f0e8d923ae42f7105bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:19:35.264081 5482 raft_consensus.cc:504] T b3b61c76bd9c443586b463de9464f811 P ba5667d185cb4f0e8d923ae42f7105bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b229b3bf9b84796bb23853b0600c783" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42595 } } peers { permanent_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34589 } } peers { permanent_uuid: "ae62d02eff7749979f0c98a8907c918a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38807 } }
I20260928 20:19:35.264479 5482 leader_election.cc:288] T b3b61c76bd9c443586b463de9464f811 P ba5667d185cb4f0e8d923ae42f7105bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b229b3bf9b84796bb23853b0600c783 (127.3.18.1:42595), ae62d02eff7749979f0c98a8907c918a (127.3.18.3:38807)
I20260928 20:19:35.268018 5485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:35.264901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:40109 (local address 127.3.18.1:42595)
0928 20:19:35.264951 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:35.264953 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:35.265023 (+ 70us) server_negotiation.cc:407] Connection header received
0928 20:19:35.265100 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:35.265101 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:35.265112 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:35.265168 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:19:35.265389 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:35.266024 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:35.267207 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:35.267604 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:35.267689 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:35.267703 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:35.267898 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 20:19:35.267936 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 20:19:35.268028 5494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:35.264830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:60805 (local address 127.3.18.3:38807)
0928 20:19:35.265023 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:35.265026 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:35.265034 (+ 8us) server_negotiation.cc:407] Connection header received
0928 20:19:35.265066 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:35.265067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:35.265078 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:35.265170 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:19:35.265425 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:35.266114 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:35.267247 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:35.267615 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:35.267704 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:35.267717 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:35.267902 (+ 185us) server_negotiation.cc:299] Negotiation successful
0928 20:19:35.267950 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":75,"threads_started":1}
I20260928 20:19:35.268061 5493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:35.264675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.1:42595 (local address 127.3.18.2:40109)
0928 20:19:35.264935 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:35.264947 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:35.265032 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:35.265186 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:35.265187 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:35.265189 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:19:35.265304 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:35.265306 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:35.266119 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:35.266120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:35.267118 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:35.267121 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:35.267694 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:35.267696 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:35.267813 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:35.267818 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 20:19:35.267886 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 20:19:35.267937 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":3,"thread_start_us":97,"threads_started":1}
I20260928 20:19:35.268047 5492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:35.264675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.3:38807 (local address 127.3.18.2:60805)
0928 20:19:35.264892 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:35.264905 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:35.264980 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:35.265225 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:35.265226 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:35.265228 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:19:35.265323 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:35.265324 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:35.266206 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:35.266207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:35.267130 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:35.267133 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:35.267722 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:35.267724 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:35.267827 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:35.267831 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:19:35.267889 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 20:19:35.267935 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":85,"thread_start_us":100,"threads_started":1}
I20260928 20:19:35.268390 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3b61c76bd9c443586b463de9464f811" candidate_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b229b3bf9b84796bb23853b0600c783" is_pre_election: true
I20260928 20:19:35.268501 5035 raft_consensus.cc:2454] T b3b61c76bd9c443586b463de9464f811 P 2b229b3bf9b84796bb23853b0600c783 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba5667d185cb4f0e8d923ae42f7105bf in term 0.
I20260928 20:19:35.268559 5315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3b61c76bd9c443586b463de9464f811" candidate_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae62d02eff7749979f0c98a8907c918a" is_pre_election: true
I20260928 20:19:35.268675 5315 raft_consensus.cc:2454] T b3b61c76bd9c443586b463de9464f811 P ae62d02eff7749979f0c98a8907c918a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba5667d185cb4f0e8d923ae42f7105bf in term 0.
I20260928 20:19:35.268687 5109 leader_election.cc:302] T b3b61c76bd9c443586b463de9464f811 P ba5667d185cb4f0e8d923ae42f7105bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b229b3bf9b84796bb23853b0600c783, ba5667d185cb4f0e8d923ae42f7105bf; no voters:
I20260928 20:19:35.268908 5482 raft_consensus.cc:2781] T b3b61c76bd9c443586b463de9464f811 P ba5667d185cb4f0e8d923ae42f7105bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 20:19:35.268953 5482 raft_consensus.cc:482] T b3b61c76bd9c443586b463de9464f811 P ba5667d185cb4f0e8d923ae42f7105bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 20:19:35.269006 5482 raft_consensus.cc:3037] T b3b61c76bd9c443586b463de9464f811 P ba5667d185cb4f0e8d923ae42f7105bf [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:35.269693 5482 raft_consensus.cc:504] T b3b61c76bd9c443586b463de9464f811 P ba5667d185cb4f0e8d923ae42f7105bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b229b3bf9b84796bb23853b0600c783" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42595 } } peers { permanent_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34589 } } peers { permanent_uuid: "ae62d02eff7749979f0c98a8907c918a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38807 } }
I20260928 20:19:35.269840 5482 leader_election.cc:288] T b3b61c76bd9c443586b463de9464f811 P ba5667d185cb4f0e8d923ae42f7105bf [CANDIDATE]: Term 1 election: Requested vote from peers 2b229b3bf9b84796bb23853b0600c783 (127.3.18.1:42595), ae62d02eff7749979f0c98a8907c918a (127.3.18.3:38807)
I20260928 20:19:35.269973 5035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3b61c76bd9c443586b463de9464f811" candidate_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b229b3bf9b84796bb23853b0600c783"
I20260928 20:19:35.269987 5315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3b61c76bd9c443586b463de9464f811" candidate_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae62d02eff7749979f0c98a8907c918a"
I20260928 20:19:35.270048 5035 raft_consensus.cc:3037] T b3b61c76bd9c443586b463de9464f811 P 2b229b3bf9b84796bb23853b0600c783 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:35.270056 5315 raft_consensus.cc:3037] T b3b61c76bd9c443586b463de9464f811 P ae62d02eff7749979f0c98a8907c918a [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:35.270622 5035 raft_consensus.cc:2454] T b3b61c76bd9c443586b463de9464f811 P 2b229b3bf9b84796bb23853b0600c783 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba5667d185cb4f0e8d923ae42f7105bf in term 1.
I20260928 20:19:35.270632 5315 raft_consensus.cc:2454] T b3b61c76bd9c443586b463de9464f811 P ae62d02eff7749979f0c98a8907c918a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba5667d185cb4f0e8d923ae42f7105bf in term 1.
I20260928 20:19:35.270788 5109 leader_election.cc:302] T b3b61c76bd9c443586b463de9464f811 P ba5667d185cb4f0e8d923ae42f7105bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b229b3bf9b84796bb23853b0600c783, ba5667d185cb4f0e8d923ae42f7105bf; no voters:
I20260928 20:19:35.270890 5482 raft_consensus.cc:2781] T b3b61c76bd9c443586b463de9464f811 P ba5667d185cb4f0e8d923ae42f7105bf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:19:35.271076 5482 raft_consensus.cc:686] T b3b61c76bd9c443586b463de9464f811 P ba5667d185cb4f0e8d923ae42f7105bf [term 1 LEADER]: Becoming Leader. State: Replica: ba5667d185cb4f0e8d923ae42f7105bf, State: Running, Role: LEADER
I20260928 20:19:35.271139 5482 consensus_queue.cc:237] T b3b61c76bd9c443586b463de9464f811 P ba5667d185cb4f0e8d923ae42f7105bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b229b3bf9b84796bb23853b0600c783" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42595 } } peers { permanent_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34589 } } peers { permanent_uuid: "ae62d02eff7749979f0c98a8907c918a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38807 } }
I20260928 20:19:35.271940 4878 catalog_manager.cc:5495] T b3b61c76bd9c443586b463de9464f811 P ba5667d185cb4f0e8d923ae42f7105bf reported cstate change: term changed from 0 to 1, leader changed from <none> to ba5667d185cb4f0e8d923ae42f7105bf (127.3.18.2). New cstate: current_term: 1 leader_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b229b3bf9b84796bb23853b0600c783" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba5667d185cb4f0e8d923ae42f7105bf" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae62d02eff7749979f0c98a8907c918a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38807 } health_report { overall_health: UNKNOWN } } }
I20260928 20:19:35.288755 5497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:35.286021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49996 (local address 127.3.18.2:34589)
0928 20:19:35.286238 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:35.286240 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:35.286246 (+ 6us) server_negotiation.cc:407] Connection header received
0928 20:19:35.286280 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:35.286280 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:35.286298 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:35.286389 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:19:35.286499 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:35.287048 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:35.287873 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:35.288101 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:35.288183 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:35.288377 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:19:35.288489 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:19:35.288573 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:35.288616 (+ 43us) server_negotiation.cc:299] Negotiation successful
0928 20:19:35.288636 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":98,"threads_started":1}
I20260928 20:19:35.294471 5500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:35.291821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.3:38807 (local address 127.3.18.2:53077)
0928 20:19:35.291978 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:35.291990 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:35.292057 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:35.292214 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:35.292215 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:35.292216 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 20:19:35.292279 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:35.292280 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:35.292892 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:35.292893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:35.293510 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:35.293512 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:35.294009 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:35.294010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:35.294073 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:35.294094 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 20:19:35.294291 (+ 197us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 20:19:35.294294 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 20:19:35.294342 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 20:19:35.294386 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":88,"threads_started":1}
I20260928 20:19:35.294443 5494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:35.291991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:53077 (local address 127.3.18.3:38807)
0928 20:19:35.292038 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:35.292040 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:35.292063 (+ 23us) server_negotiation.cc:407] Connection header received
0928 20:19:35.292122 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:35.292123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:35.292136 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:35.292205 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:19:35.292361 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:35.292815 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:35.293602 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:35.293911 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:35.293991 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:35.294186 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:19:35.294220 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:19:35.294288 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:35.294344 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 20:19:35.294369 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 20:19:35.296829 5315 raft_consensus.cc:1261] T b3b61c76bd9c443586b463de9464f811 P ae62d02eff7749979f0c98a8907c918a [term 1 FOLLOWER]: Refusing update from remote peer ba5667d185cb4f0e8d923ae42f7105bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:19:35.296931 5035 raft_consensus.cc:1261] T b3b61c76bd9c443586b463de9464f811 P 2b229b3bf9b84796bb23853b0600c783 [term 1 FOLLOWER]: Refusing update from remote peer ba5667d185cb4f0e8d923ae42f7105bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:19:35.297119 5495 consensus_queue.cc:1035] T b3b61c76bd9c443586b463de9464f811 P ba5667d185cb4f0e8d923ae42f7105bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae62d02eff7749979f0c98a8907c918a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:19:35.297286 5482 consensus_queue.cc:1035] T b3b61c76bd9c443586b463de9464f811 P ba5667d185cb4f0e8d923ae42f7105bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b229b3bf9b84796bb23853b0600c783" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:19:35.304504 5497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:35.302002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:47991 (local address 127.3.18.2:34589)
0928 20:19:35.302043 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:35.302045 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:35.302169 (+ 124us) server_negotiation.cc:407] Connection header received
0928 20:19:35.302208 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:35.302209 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:35.302219 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:35.302280 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:19:35.302392 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:35.302899 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:35.303788 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:35.304037 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:35.304120 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:35.304261 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:19:35.304295 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:19:35.304344 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:35.304409 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 20:19:35.304431 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 20:19:35.304520 5505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:35.301968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.2:34589 (local address 127.3.18.3:47991)
0928 20:19:35.302094 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:35.302105 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:35.302161 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:35.302281 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:35.302282 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:35.302283 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 20:19:35.302336 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:35.302338 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:35.303010 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:35.303011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:35.303666 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:35.303669 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:35.304123 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:35.304124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:35.304184 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:35.304189 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 20:19:35.304354 (+ 165us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 20:19:35.304356 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 20:19:35.304404 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 20:19:35.304448 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":84,"threads_started":1}
W20260928 20:19:35.304898 5448 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 20:19:36.828385 4877 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60448:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:19:36.828512 4877 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60448:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:19:36.829692 4877 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cddefad8370040909095f66cc0817b0e: Sending DeleteTablet for 3 replicas of tablet b3b61c76bd9c443586b463de9464f811
I20260928 20:19:36.830010 5015 tablet_service.cc:1514] Processing DeleteTablet for tablet b3b61c76bd9c443586b463de9464f811 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:19:36 UTC) from {username='kudu', principal='kudu/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:42410
I20260928 20:19:36.830051 5155 tablet_service.cc:1514] Processing DeleteTablet for tablet b3b61c76bd9c443586b463de9464f811 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:19:36 UTC) from {username='kudu', principal='kudu/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:49984
I20260928 20:19:36.830205 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 4950
I20260928 20:19:36.830776 5294 tablet_service.cc:1514] Processing DeleteTablet for tablet b3b61c76bd9c443586b463de9464f811 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:19:36 UTC) from {username='kudu', principal='kudu/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:48108
I20260928 20:19:36.830996 5532 tablet_replica.cc:330] T b3b61c76bd9c443586b463de9464f811 P ae62d02eff7749979f0c98a8907c918a: stopping tablet replica
I20260928 20:19:36.831072 5532 raft_consensus.cc:2229] T b3b61c76bd9c443586b463de9464f811 P ae62d02eff7749979f0c98a8907c918a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:19:36.831118 5532 raft_consensus.cc:2258] T b3b61c76bd9c443586b463de9464f811 P ae62d02eff7749979f0c98a8907c918a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:19:36.831370 5532 ts_tablet_manager.cc:1905] T b3b61c76bd9c443586b463de9464f811 P ae62d02eff7749979f0c98a8907c918a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:19:36.833335 5532 ts_tablet_manager.cc:1918] T b3b61c76bd9c443586b463de9464f811 P ae62d02eff7749979f0c98a8907c918a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 20:19:36.833410 5532 log.cc:1196] T b3b61c76bd9c443586b463de9464f811 P ae62d02eff7749979f0c98a8907c918a: Deleting WAL directory at /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/ts-2/wal/wals/b3b61c76bd9c443586b463de9464f811
I20260928 20:19:36.833632 5532 ts_tablet_manager.cc:1939] T b3b61c76bd9c443586b463de9464f811 P ae62d02eff7749979f0c98a8907c918a: Deleting consensus metadata
I20260928 20:19:36.834307 4863 catalog_manager.cc:4841] TS ae62d02eff7749979f0c98a8907c918a (127.3.18.3:38807): tablet b3b61c76bd9c443586b463de9464f811 (table test-table [id=949b88ee3c964c3caff085bdc374c21a]) successfully deleted
I20260928 20:19:36.835932 5531 tablet_replica.cc:330] T b3b61c76bd9c443586b463de9464f811 P ba5667d185cb4f0e8d923ae42f7105bf: stopping tablet replica
I20260928 20:19:36.836009 5531 raft_consensus.cc:2229] T b3b61c76bd9c443586b463de9464f811 P ba5667d185cb4f0e8d923ae42f7105bf [term 1 LEADER]: Raft consensus shutting down.
I20260928 20:19:36.836158 5531 raft_consensus.cc:2258] T b3b61c76bd9c443586b463de9464f811 P ba5667d185cb4f0e8d923ae42f7105bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:19:36.836475 5531 ts_tablet_manager.cc:1905] T b3b61c76bd9c443586b463de9464f811 P ba5667d185cb4f0e8d923ae42f7105bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:19:36.838060 5531 ts_tablet_manager.cc:1918] T b3b61c76bd9c443586b463de9464f811 P ba5667d185cb4f0e8d923ae42f7105bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 20:19:36.838116 5531 log.cc:1196] T b3b61c76bd9c443586b463de9464f811 P ba5667d185cb4f0e8d923ae42f7105bf: Deleting WAL directory at /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790626764250411-3144-0/minicluster-data/ts-1/wal/wals/b3b61c76bd9c443586b463de9464f811
W20260928 20:19:36.838130 4865 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.18.1:42595 (error 108)
I20260928 20:19:36.838394 5531 ts_tablet_manager.cc:1939] T b3b61c76bd9c443586b463de9464f811 P ba5667d185cb4f0e8d923ae42f7105bf: Deleting consensus metadata
I20260928 20:19:36.838817 4864 catalog_manager.cc:4841] TS ba5667d185cb4f0e8d923ae42f7105bf (127.3.18.2:34589): tablet b3b61c76bd9c443586b463de9464f811 (table test-table [id=949b88ee3c964c3caff085bdc374c21a]) successfully deleted
W20260928 20:19:36.838891 5534 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:19:36.838601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.1:42595 (local address 127.0.0.1:42446)
0928 20:19:36.838757 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:36.838770 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:36.838835 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.1:42595: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":79,"threads_started":1}
W20260928 20:19:36.839006 4865 catalog_manager.cc:4584] TS 2b229b3bf9b84796bb23853b0600c783 (127.3.18.1:42595): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b3b61c76bd9c443586b463de9464f811: Network error: Client connection negotiation failed: client connection to 127.3.18.1:42595: BlockingWrite error: write error: Connection reset by peer (error 104)
I20260928 20:19:36.839208 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 5090
I20260928 20:19:36.845937 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 5230
I20260928 20:19:36.851990 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 4846
2026-09-28T20:19:36Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6120 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:19:36 dist-test-slave-b83v krb5kdc[5537](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:19:36 dist-test-slave-b83v krb5kdc[5537](info): set up 2 sockets
Sep 28 20:19:36 dist-test-slave-b83v krb5kdc[5537](info): commencing operation
krb5kdc: starting...
W20260928 20:19:37.927951 3144 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Sep 28 20:19:38 dist-test-slave-b83v krb5kdc[5537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:19:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:19:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.18.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab.
Entry for principal kudu/127.3.18.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab.
I20260928 20:19:38.055020 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:45881
--webserver_interface=127.3.18.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:34541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.18.62:45881
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:19:38.141326 5550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:19:38.141670 5550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:19:38.141716 5550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:19:38.143390 5550 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:19:38.143445 5550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:19:38.143460 5550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:19:38.143474 5550 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:19:38.143486 5550 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:19:38.145896 5550 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:34541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.18.62:45881
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:45881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.18.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:19:38.146433 5550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:19:38.146734 5550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:19:38.153154 5556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:38.153157 5558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:38.153174 5555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:38.153517 5550 server_base.cc:1034] running on GCE node
I20260928 20:19:38.153790 5550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:19:38.154130 5550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:19:38.155359 5550 hybrid_clock.cc:648] HybridClock initialized: now 1790626778155305 us; error 59 us; skew 500 ppm
Sep 28 20:19:38 dist-test-slave-b83v krb5kdc[5537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626778, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:19:38.158330 5550 init.cc:377] Logged in from keytab as kudu/127.3.18.62@KRBTEST.COM (short username kudu)
I20260928 20:19:38.158983 5550 webserver.cc:460] Webserver started at http://127.3.18.62:39709/ using document root <none> and password file <none>
I20260928 20:19:38.159253 5550 fs_manager.cc:359] Metadata directory not provided
I20260928 20:19:38.159323 5550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:19:38.159488 5550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:19:38.160710 5550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/minicluster-data/master-0/data/instance:
uuid: "49d2b4b62f1540f28c6664b597b77068"
format_stamp: "Formatted at 2026-09-28 20:19:38 on dist-test-slave-b83v"
server_key: "f557e1104e1fa5848179216a56586ed3"
server_key_iv: "e839d8e558c1666f4eb393938d721294"
server_key_version: "encryptionkey@0"
I20260928 20:19:38.161121 5550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/minicluster-data/master-0/wal/instance:
uuid: "49d2b4b62f1540f28c6664b597b77068"
format_stamp: "Formatted at 2026-09-28 20:19:38 on dist-test-slave-b83v"
server_key: "f557e1104e1fa5848179216a56586ed3"
server_key_iv: "e839d8e558c1666f4eb393938d721294"
server_key_version: "encryptionkey@0"
I20260928 20:19:38.162626 5550 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 20:19:38.163645 5565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:38.163939 5550 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:19:38.164060 5550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/minicluster-data/master-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/minicluster-data/master-0/wal
uuid: "49d2b4b62f1540f28c6664b597b77068"
format_stamp: "Formatted at 2026-09-28 20:19:38 on dist-test-slave-b83v"
server_key: "f557e1104e1fa5848179216a56586ed3"
server_key_iv: "e839d8e558c1666f4eb393938d721294"
server_key_version: "encryptionkey@0"
I20260928 20:19:38.164167 5550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:19:38.175525 5550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:19:38.178399 5550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:19:38.178608 5550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:19:38.183374 5550 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.62:45881
I20260928 20:19:38.183408 5617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.62:45881 every 8 connection(s)
I20260928 20:19:38.183898 5550 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
I20260928 20:19:38.184896 5618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:38.187614 5618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49d2b4b62f1540f28c6664b597b77068: Bootstrap starting.
I20260928 20:19:38.188371 5618 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49d2b4b62f1540f28c6664b597b77068: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:38.188735 5618 log.cc:824] T 00000000000000000000000000000000 P 49d2b4b62f1540f28c6664b597b77068: Log is configured to *not* fsync() on all Append() calls
I20260928 20:19:38.189678 5618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49d2b4b62f1540f28c6664b597b77068: No bootstrap required, opened a new log
I20260928 20:19:38.190322 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 5550
I20260928 20:19:38.190483 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/minicluster-data/master-0/wal/instance
I20260928 20:19:38.190675 3144 external_mini_cluster.cc:1428] Setting key df7dcb3a64358faeab530b407c7244f9
I20260928 20:19:38.192174 5618 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49d2b4b62f1540f28c6664b597b77068 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49d2b4b62f1540f28c6664b597b77068" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 45881 } }
I20260928 20:19:38.192366 5618 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49d2b4b62f1540f28c6664b597b77068 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:38.192405 5618 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49d2b4b62f1540f28c6664b597b77068 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49d2b4b62f1540f28c6664b597b77068, State: Initialized, Role: FOLLOWER
I20260928 20:19:38.192570 5618 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49d2b4b62f1540f28c6664b597b77068 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49d2b4b62f1540f28c6664b597b77068" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 45881 } }
I20260928 20:19:38.192683 5618 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49d2b4b62f1540f28c6664b597b77068 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:19:38.192718 5618 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49d2b4b62f1540f28c6664b597b77068 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:19:38.192771 5618 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49d2b4b62f1540f28c6664b597b77068 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:38.193917 5618 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49d2b4b62f1540f28c6664b597b77068 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49d2b4b62f1540f28c6664b597b77068" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 45881 } }
I20260928 20:19:38.194026 5618 leader_election.cc:302] T 00000000000000000000000000000000 P 49d2b4b62f1540f28c6664b597b77068 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49d2b4b62f1540f28c6664b597b77068; no voters:
I20260928 20:19:38.194275 5618 leader_election.cc:288] T 00000000000000000000000000000000 P 49d2b4b62f1540f28c6664b597b77068 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:19:38.194314 5623 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49d2b4b62f1540f28c6664b597b77068 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:19:38.194425 5623 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49d2b4b62f1540f28c6664b597b77068 [term 1 LEADER]: Becoming Leader. State: Replica: 49d2b4b62f1540f28c6664b597b77068, State: Running, Role: LEADER
I20260928 20:19:38.194581 5618 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49d2b4b62f1540f28c6664b597b77068 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:19:38.194583 5623 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49d2b4b62f1540f28c6664b597b77068 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49d2b4b62f1540f28c6664b597b77068" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 45881 } }
I20260928 20:19:38.195164 5625 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49d2b4b62f1540f28c6664b597b77068 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49d2b4b62f1540f28c6664b597b77068. Latest consensus state: current_term: 1 leader_uuid: "49d2b4b62f1540f28c6664b597b77068" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49d2b4b62f1540f28c6664b597b77068" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 45881 } } }
I20260928 20:19:38.195261 5625 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49d2b4b62f1540f28c6664b597b77068 [sys.catalog]: This master's current role is: LEADER
I20260928 20:19:38.195501 5624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49d2b4b62f1540f28c6664b597b77068 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49d2b4b62f1540f28c6664b597b77068" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49d2b4b62f1540f28c6664b597b77068" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 45881 } } }
I20260928 20:19:38.195879 5624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49d2b4b62f1540f28c6664b597b77068 [sys.catalog]: This master's current role is: LEADER
I20260928 20:19:38.196066 5630 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:19:38.196870 5630 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 20:19:38 dist-test-slave-b83v krb5kdc[5537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:19:38.198694 5630 catalog_manager.cc:1347] Generated new cluster ID: 54fa8f85c2604e4b845aaff48165c22c
I20260928 20:19:38.198752 5630 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:19:38.203069 5622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:38.191306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59954 (local address 127.3.18.62:45881)
0928 20:19:38.191511 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:38.191525 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:38.191555 (+ 30us) server_negotiation.cc:407] Connection header received
0928 20:19:38.192144 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:38.192152 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:38.192221 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:38.192488 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:38.193301 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:38.194438 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:38.195502 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:38.196213 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:38.196833 (+ 620us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:38.199208 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:38.199212 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:38.199213 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:38.199225 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:38.201091 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:38.201493 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:38.201497 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:38.201497 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:38.201598 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:38.201814 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:38.201814 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:38.201815 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:38.202060 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:38.202724 (+ 664us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:38.202791 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 20:19:38.202919 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":112,"threads_started":1}
I20260928 20:19:38.213363 5630 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:19:38.214111 5630 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:19:38.216934 5630 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49d2b4b62f1540f28c6664b597b77068: Generated new TSK 0
I20260928 20:19:38.217154 5630 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.18.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab.
Entry for principal kudu/127.3.18.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab.
I20260928 20:19:38.251037 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.1:0
--local_ip_for_outbound_sockets=127.3.18.1
--webserver_interface=127.3.18.1
--webserver_port=0
--tserver_master_addrs=127.3.18.62:45881
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab
--principal=kudu/127.3.18.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:34541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:19:38.336666 5644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:19:38.336891 5644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:19:38.336916 5644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:19:38.338105 5644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:19:38.338146 5644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:19:38.338172 5644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.1
I20260928 20:19:38.339635 5644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:34541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab
--principal=kudu/127.3.18.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.18.1
--webserver_port=0
--tserver_master_addrs=127.3.18.62:45881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.1
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:19:38.339948 5644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:19:38.340158 5644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:19:38.345667 5649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:38.345667 5652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:38.345758 5650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:38.345687 5644 server_base.cc:1034] running on GCE node
I20260928 20:19:38.346138 5644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:19:38.346431 5644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:19:38.347604 5644 hybrid_clock.cc:648] HybridClock initialized: now 1790626778347583 us; error 38 us; skew 500 ppm
Sep 28 20:19:38 dist-test-slave-b83v krb5kdc[5537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626778, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:19:38.350322 5644 init.cc:377] Logged in from keytab as kudu/127.3.18.1@KRBTEST.COM (short username kudu)
I20260928 20:19:38.350867 5644 webserver.cc:460] Webserver started at http://127.3.18.1:36907/ using document root <none> and password file <none>
I20260928 20:19:38.351130 5644 fs_manager.cc:359] Metadata directory not provided
I20260928 20:19:38.351181 5644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:19:38.351358 5644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:19:38.352584 5644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-0/data/instance:
uuid: "039189669a40430dbc6770fd379ede1f"
format_stamp: "Formatted at 2026-09-28 20:19:38 on dist-test-slave-b83v"
server_key: "bd0d539b0b6025d0cc10503d4f058488"
server_key_iv: "aba344128c06801a0263ddd751cae1f1"
server_key_version: "encryptionkey@0"
I20260928 20:19:38.353001 5644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-0/wal/instance:
uuid: "039189669a40430dbc6770fd379ede1f"
format_stamp: "Formatted at 2026-09-28 20:19:38 on dist-test-slave-b83v"
server_key: "bd0d539b0b6025d0cc10503d4f058488"
server_key_iv: "aba344128c06801a0263ddd751cae1f1"
server_key_version: "encryptionkey@0"
I20260928 20:19:38.354496 5644 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 20:19:38.355379 5659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:38.355630 5644 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:19:38.355741 5644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-0/wal
uuid: "039189669a40430dbc6770fd379ede1f"
format_stamp: "Formatted at 2026-09-28 20:19:38 on dist-test-slave-b83v"
server_key: "bd0d539b0b6025d0cc10503d4f058488"
server_key_iv: "aba344128c06801a0263ddd751cae1f1"
server_key_version: "encryptionkey@0"
I20260928 20:19:38.355897 5644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:19:38.367141 5644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:19:38.369820 5644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:19:38.370007 5644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:19:38.370265 5644 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:19:38.370620 5644 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:19:38.370685 5644 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:38.370723 5644 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:19:38.370776 5644 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:38.378388 5644 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.1:40617
I20260928 20:19:38.378427 5772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.1:40617 every 8 connection(s)
I20260928 20:19:38.378790 5644 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
Sep 28 20:19:38 dist-test-slave-b83v krb5kdc[5537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626778, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.1@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:19:38.386617 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 5644
I20260928 20:19:38.386744 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-0/wal/instance
I20260928 20:19:38.387032 3144 external_mini_cluster.cc:1428] Setting key 972779b1214a0ffae63a7a17652faea2
I20260928 20:19:38.389330 5622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:38.379560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:32887 (local address 127.3.18.62:45881)
0928 20:19:38.379617 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:38.379619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:38.379882 (+ 263us) server_negotiation.cc:407] Connection header received
0928 20:19:38.380420 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:38.380421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:38.380437 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:38.380489 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:38.381566 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:38.382066 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:38.383091 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:38.383308 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:38.383394 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:38.385921 (+ 2527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:38.385923 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:38.385923 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:38.385932 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:38.387700 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:38.388222 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:38.388224 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:38.388225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:38.388303 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:38.388581 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:38.388582 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:38.388582 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:38.388718 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:38.388805 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:38.389108 (+ 303us) server_negotiation.cc:299] Negotiation successful
0928 20:19:38.389214 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 20:19:38.389573 5775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:38.379530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:45881 (local address 127.3.18.1:32887)
0928 20:19:38.379708 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:38.379747 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:38.380304 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:38.380578 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:38.380581 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:38.381239 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:38.381462 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:38.381467 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:38.382186 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:38.382187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:38.382987 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:38.382990 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:38.383406 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:38.383407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:38.383537 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:38.384202 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:19:38.384211 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:38.385816 (+ 1605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:38.387845 (+ 2029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:38.387854 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:38.387856 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:38.388118 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:38.388379 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:38.388380 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:38.388380 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:38.388504 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:38.388796 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:38.388803 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:38.388965 (+ 162us) client_negotiation.cc:769] Sending connection context
0928 20:19:38.389086 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 20:19:38.389264 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":93,"threads_started":1}
I20260928 20:19:38.390165 5773 heartbeater.cc:347] Connected to a master server at 127.3.18.62:45881
I20260928 20:19:38.390313 5773 heartbeater.cc:464] Registering TS with master...
I20260928 20:19:38.390517 5773 heartbeater.cc:511] Master 127.3.18.62:45881 requested a full tablet report, sending...
I20260928 20:19:38.391084 5582 ts_manager.cc:194] Registered new tserver with Master: 039189669a40430dbc6770fd379ede1f (127.3.18.1:40617)
I20260928 20:19:38.392104 5582 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.18.1@KRBTEST.COM'} at 127.3.18.1:32887
WARNING: no policy specified for kudu/127.3.18.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab.
Entry for principal kudu/127.3.18.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab.
I20260928 20:19:38.415369 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.2:0
--local_ip_for_outbound_sockets=127.3.18.2
--webserver_interface=127.3.18.2
--webserver_port=0
--tserver_master_addrs=127.3.18.62:45881
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab
--principal=kudu/127.3.18.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:34541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:19:38.497386 5778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:19:38.497634 5778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:19:38.497670 5778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:19:38.498867 5778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:19:38.498947 5778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:19:38.498991 5778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.2
I20260928 20:19:38.500617 5778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:34541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab
--principal=kudu/127.3.18.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.18.2
--webserver_port=0
--tserver_master_addrs=127.3.18.62:45881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.2
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:19:38.500950 5778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:19:38.501216 5778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:19:38.507375 5783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:38.507454 5778 server_base.cc:1034] running on GCE node
W20260928 20:19:38.507411 5784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:38.507375 5786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:38.507977 5778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:19:38.508322 5778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:19:38.509500 5778 hybrid_clock.cc:648] HybridClock initialized: now 1790626778509474 us; error 37 us; skew 500 ppm
Sep 28 20:19:38 dist-test-slave-b83v krb5kdc[5537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626778, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:19:38.512480 5778 init.cc:377] Logged in from keytab as kudu/127.3.18.2@KRBTEST.COM (short username kudu)
I20260928 20:19:38.512967 5778 webserver.cc:460] Webserver started at http://127.3.18.2:46537/ using document root <none> and password file <none>
I20260928 20:19:38.513190 5778 fs_manager.cc:359] Metadata directory not provided
I20260928 20:19:38.513242 5778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:19:38.513343 5778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:19:38.514482 5778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-1/data/instance:
uuid: "5a6453388cf240e6bd457409de199e27"
format_stamp: "Formatted at 2026-09-28 20:19:38 on dist-test-slave-b83v"
server_key: "734b4b6cb39a6fb9ae2627bf1b181a63"
server_key_iv: "af3752925a6a1b574afab475c945e382"
server_key_version: "encryptionkey@0"
I20260928 20:19:38.514829 5778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-1/wal/instance:
uuid: "5a6453388cf240e6bd457409de199e27"
format_stamp: "Formatted at 2026-09-28 20:19:38 on dist-test-slave-b83v"
server_key: "734b4b6cb39a6fb9ae2627bf1b181a63"
server_key_iv: "af3752925a6a1b574afab475c945e382"
server_key_version: "encryptionkey@0"
I20260928 20:19:38.516441 5778 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:19:38.517385 5793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:38.517608 5778 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:19:38.517733 5778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-1/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-1/wal
uuid: "5a6453388cf240e6bd457409de199e27"
format_stamp: "Formatted at 2026-09-28 20:19:38 on dist-test-slave-b83v"
server_key: "734b4b6cb39a6fb9ae2627bf1b181a63"
server_key_iv: "af3752925a6a1b574afab475c945e382"
server_key_version: "encryptionkey@0"
I20260928 20:19:38.517823 5778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:19:38.529150 5778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:19:38.531729 5778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:19:38.532011 5778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:19:38.532307 5778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:19:38.532662 5778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:19:38.532717 5778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:38.532766 5778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:19:38.532809 5778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:38.538671 5778 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.2:45993
I20260928 20:19:38.538743 5906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.2:45993 every 8 connection(s)
I20260928 20:19:38.539106 5778 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
I20260928 20:19:38.540505 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 5778
I20260928 20:19:38.540589 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-1/wal/instance
I20260928 20:19:38.540791 3144 external_mini_cluster.cc:1428] Setting key 5961614699b04593840c0d9531323049
Sep 28 20:19:38 dist-test-slave-b83v krb5kdc[5537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626778, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.2@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.18.3@KRBTEST.COM; defaulting to no policy
I20260928 20:19:38.550148 5622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:38.539927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:60667 (local address 127.3.18.62:45881)
0928 20:19:38.540003 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:38.540033 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:38.540234 (+ 201us) server_negotiation.cc:407] Connection header received
0928 20:19:38.540845 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:38.540846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:38.540860 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:38.540922 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:38.542154 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:38.542672 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:38.543743 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:38.544029 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:38.544121 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:38.546942 (+ 2821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:38.546944 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:38.546944 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:38.546952 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:38.548520 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:38.549030 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:38.549031 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:38.549031 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:38.549085 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:38.549418 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:38.549418 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:38.549419 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:38.549528 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:38.549582 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:38.549932 (+ 350us) server_negotiation.cc:299] Negotiation successful
0928 20:19:38.550018 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 20:19:38.550438 5909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:38.539888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:45881 (local address 127.3.18.2:60667)
0928 20:19:38.540106 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:38.540144 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:38.540730 (+ 586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:38.541007 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:38.541011 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:38.541846 (+ 835us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:38.542051 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:38.542056 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:38.542770 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:38.542771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:38.543623 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:38.543625 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:38.544131 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:38.544133 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:38.544283 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:38.544880 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:19:38.544892 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:38.546800 (+ 1908us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:38.548630 (+ 1830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:38.548638 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:38.548641 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:38.548950 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:38.549161 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:38.549162 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:38.549163 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:38.549343 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:38.549595 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:38.549602 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:38.549795 (+ 193us) client_negotiation.cc:769] Sending connection context
0928 20:19:38.549915 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 20:19:38.550124 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":102,"threads_started":1}
I20260928 20:19:38.551034 5907 heartbeater.cc:347] Connected to a master server at 127.3.18.62:45881
I20260928 20:19:38.551298 5907 heartbeater.cc:464] Registering TS with master...
I20260928 20:19:38.551527 5907 heartbeater.cc:511] Master 127.3.18.62:45881 requested a full tablet report, sending...
I20260928 20:19:38.552158 5582 ts_manager.cc:194] Registered new tserver with Master: 5a6453388cf240e6bd457409de199e27 (127.3.18.2:45993)
I20260928 20:19:38.552606 5582 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.18.2@KRBTEST.COM'} at 127.3.18.2:60667
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab.
Entry for principal kudu/127.3.18.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab.
I20260928 20:19:38.572750 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.3:0
--local_ip_for_outbound_sockets=127.3.18.3
--webserver_interface=127.3.18.3
--webserver_port=0
--tserver_master_addrs=127.3.18.62:45881
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab
--principal=kudu/127.3.18.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:34541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:19:38.657414 5912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:19:38.657682 5912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:19:38.657725 5912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:19:38.658991 5912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:19:38.659065 5912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:19:38.659116 5912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.3
I20260928 20:19:38.660840 5912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:34541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab
--principal=kudu/127.3.18.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.18.3
--webserver_port=0
--tserver_master_addrs=127.3.18.62:45881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.3
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:19:38.661124 5912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:19:38.661377 5912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 20:19:38.666709 5912 server_base.cc:1034] running on GCE node
W20260928 20:19:38.666754 5917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:38.666854 5918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:38.666690 5920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:38.667198 5912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:19:38.667467 5912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:19:38.668625 5912 hybrid_clock.cc:648] HybridClock initialized: now 1790626778668614 us; error 33 us; skew 500 ppm
Sep 28 20:19:38 dist-test-slave-b83v krb5kdc[5537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626778, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:19:38.671370 5912 init.cc:377] Logged in from keytab as kudu/127.3.18.3@KRBTEST.COM (short username kudu)
I20260928 20:19:38.672003 5912 webserver.cc:460] Webserver started at http://127.3.18.3:44419/ using document root <none> and password file <none>
I20260928 20:19:38.672263 5912 fs_manager.cc:359] Metadata directory not provided
I20260928 20:19:38.672310 5912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:19:38.672467 5912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:19:38.673569 5912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-2/data/instance:
uuid: "39e1e3c1c0b348fc8d0ba31d3bd1bf0c"
format_stamp: "Formatted at 2026-09-28 20:19:38 on dist-test-slave-b83v"
server_key: "0dbcb80a465fcabe5256c748df929a88"
server_key_iv: "3a3bc2052cd6a3119ff5574d5eb3b6be"
server_key_version: "encryptionkey@0"
I20260928 20:19:38.673935 5912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-2/wal/instance:
uuid: "39e1e3c1c0b348fc8d0ba31d3bd1bf0c"
format_stamp: "Formatted at 2026-09-28 20:19:38 on dist-test-slave-b83v"
server_key: "0dbcb80a465fcabe5256c748df929a88"
server_key_iv: "3a3bc2052cd6a3119ff5574d5eb3b6be"
server_key_version: "encryptionkey@0"
I20260928 20:19:38.675385 5912 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 20:19:38.676460 5927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:38.676692 5912 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 20:19:38.676801 5912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-2/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-2/wal
uuid: "39e1e3c1c0b348fc8d0ba31d3bd1bf0c"
format_stamp: "Formatted at 2026-09-28 20:19:38 on dist-test-slave-b83v"
server_key: "0dbcb80a465fcabe5256c748df929a88"
server_key_iv: "3a3bc2052cd6a3119ff5574d5eb3b6be"
server_key_version: "encryptionkey@0"
I20260928 20:19:38.676892 5912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:19:38.685851 5912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:19:38.688012 5912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:19:38.688181 5912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:19:38.688457 5912 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:19:38.688831 5912 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:19:38.688894 5912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:38.688934 5912 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:19:38.688962 5912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:38.694957 5912 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.3:43663
I20260928 20:19:38.695006 6040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.3:43663 every 8 connection(s)
I20260928 20:19:38.695401 5912 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
I20260928 20:19:38.698942 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 5912
I20260928 20:19:38.699052 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-2/wal/instance
I20260928 20:19:38.699277 3144 external_mini_cluster.cc:1428] Setting key 279692206c75e094787ced62f5b8b0a2
Sep 28 20:19:38 dist-test-slave-b83v krb5kdc[5537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626778, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.3@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:19:38.706236 5622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:38.696218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:53515 (local address 127.3.18.62:45881)
0928 20:19:38.696260 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:38.696262 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:38.696536 (+ 274us) server_negotiation.cc:407] Connection header received
0928 20:19:38.697081 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:38.697082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:38.697096 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:38.697141 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:38.698157 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:38.698894 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:38.700721 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:38.701098 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:38.701179 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:38.703504 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:38.703505 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:38.703506 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:38.703514 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:38.704790 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:38.705201 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:38.705203 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:38.705203 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:38.705286 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:38.705556 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:38.705557 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:38.705558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:38.705683 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:38.705762 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:38.706045 (+ 283us) server_negotiation.cc:299] Negotiation successful
0928 20:19:38.706141 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 20:19:38.706477 6043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:38.696226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:45881 (local address 127.3.18.3:53515)
0928 20:19:38.696410 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:38.696445 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:38.696967 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:38.697231 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:38.697235 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:38.697825 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:38.698018 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:38.698023 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:38.699020 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:38.699022 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:38.700629 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:38.700633 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:38.701208 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:38.701209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:38.701349 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:38.701846 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:19:38.701854 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:38.703401 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:38.704876 (+ 1475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:38.704883 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:38.704886 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:38.705109 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:38.705365 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:38.705366 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:38.705366 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:38.705486 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:38.705764 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:38.705771 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:38.705916 (+ 145us) client_negotiation.cc:769] Sending connection context
0928 20:19:38.706026 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 20:19:38.706215 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":76,"threads_started":1}
I20260928 20:19:38.706985 6041 heartbeater.cc:347] Connected to a master server at 127.3.18.62:45881
I20260928 20:19:38.707086 6041 heartbeater.cc:464] Registering TS with master...
I20260928 20:19:38.707347 6041 heartbeater.cc:511] Master 127.3.18.62:45881 requested a full tablet report, sending...
I20260928 20:19:38.707823 5582 ts_manager.cc:194] Registered new tserver with Master: 39e1e3c1c0b348fc8d0ba31d3bd1bf0c (127.3.18.3:43663)
I20260928 20:19:38.708312 5582 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.18.3@KRBTEST.COM'} at 127.3.18.3:53515
I20260928 20:19:38.712294 3144 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 20:19:38.723552 5622 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:19:38.722500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59970 (local address 127.3.18.62:45881)
0928 20:19:38.722570 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:38.722572 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:38.722776 (+ 204us) server_negotiation.cc:407] Connection header received
0928 20:19:38.722827 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:38.722827 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:38.722837 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:38.722887 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:38.723485 (+ 598us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59970: BlockingRecv error: recv got EOF from 127.0.0.1:59970 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 20:19:38.724443 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 5644
I20260928 20:19:38.729126 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 5778
I20260928 20:19:38.733582 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 5912
I20260928 20:19:38.737850 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 5550
2026-09-28T20:19:38Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1883 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790626764250411-3144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:19:38 dist-test-slave-b83v krb5kdc[6053](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:19:38 dist-test-slave-b83v krb5kdc[6053](info): set up 2 sockets
Sep 28 20:19:38 dist-test-slave-b83v krb5kdc[6053](info): commencing operation
krb5kdc: starting...
W20260928 20:19:39.784066 3144 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Sep 28 20:19:39 dist-test-slave-b83v krb5kdc[6053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:19:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:19:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.18.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab.
Entry for principal kudu/127.3.18.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab.
I20260928 20:19:39.915846 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790626764250411-3144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790626764250411-3144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790626764250411-3144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:36163
--webserver_interface=127.3.18.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:41389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.18.62:36163
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:19:40.002831 6066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:19:40.003047 6066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:19:40.003070 6066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:19:40.004287 6066 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:19:40.004330 6066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:19:40.004344 6066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:19:40.004357 6066 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:19:40.004370 6066 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:19:40.005934 6066 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:41389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790626764250411-3144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790626764250411-3144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.18.62:36163
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:36163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.18.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790626764250411-3144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:19:40.006198 6066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:19:40.006440 6066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:19:40.011898 6072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:40.011946 6066 server_base.cc:1034] running on GCE node
W20260928 20:19:40.011924 6071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:40.011929 6074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:40.012552 6066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:19:40.012926 6066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:19:40.014104 6066 hybrid_clock.cc:648] HybridClock initialized: now 1790626780014078 us; error 49 us; skew 500 ppm
Sep 28 20:19:40 dist-test-slave-b83v krb5kdc[6053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626780, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:19:40.017073 6066 init.cc:377] Logged in from keytab as kudu/127.3.18.62@KRBTEST.COM (short username kudu)
I20260928 20:19:40.017573 6066 webserver.cc:460] Webserver started at http://127.3.18.62:45495/ using document root <none> and password file <none>
I20260928 20:19:40.017838 6066 fs_manager.cc:359] Metadata directory not provided
I20260928 20:19:40.017891 6066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:19:40.018038 6066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:19:40.019224 6066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790626764250411-3144-0/minicluster-data/master-0/data/instance:
uuid: "3bc45057888d479a8aa5e19ebe4ad468"
format_stamp: "Formatted at 2026-09-28 20:19:40 on dist-test-slave-b83v"
server_key: "258fa7271eb3a937f7d7b3f6995302be"
server_key_iv: "23bf1ffa9075a45fb2f1bafb2911ee8a"
server_key_version: "encryptionkey@0"
I20260928 20:19:40.019630 6066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790626764250411-3144-0/minicluster-data/master-0/wal/instance:
uuid: "3bc45057888d479a8aa5e19ebe4ad468"
format_stamp: "Formatted at 2026-09-28 20:19:40 on dist-test-slave-b83v"
server_key: "258fa7271eb3a937f7d7b3f6995302be"
server_key_iv: "23bf1ffa9075a45fb2f1bafb2911ee8a"
server_key_version: "encryptionkey@0"
I20260928 20:19:40.021178 6066 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:19:40.022074 6081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:40.022362 6066 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 20:19:40.022485 6066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790626764250411-3144-0/minicluster-data/master-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790626764250411-3144-0/minicluster-data/master-0/wal
uuid: "3bc45057888d479a8aa5e19ebe4ad468"
format_stamp: "Formatted at 2026-09-28 20:19:40 on dist-test-slave-b83v"
server_key: "258fa7271eb3a937f7d7b3f6995302be"
server_key_iv: "23bf1ffa9075a45fb2f1bafb2911ee8a"
server_key_version: "encryptionkey@0"
I20260928 20:19:40.022567 6066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790626764250411-3144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790626764250411-3144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790626764250411-3144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:19:40.040827 6066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:19:40.043380 6066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:19:40.043569 6066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:19:40.047611 6066 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.62:36163
I20260928 20:19:40.047647 6133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.62:36163 every 8 connection(s)
I20260928 20:19:40.048101 6066 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
I20260928 20:19:40.049042 6134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:40.051519 6134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bc45057888d479a8aa5e19ebe4ad468: Bootstrap starting.
I20260928 20:19:40.051906 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 6066
I20260928 20:19:40.052016 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790626764250411-3144-0/minicluster-data/master-0/wal/instance
I20260928 20:19:40.052209 3144 external_mini_cluster.cc:1428] Setting key 0fa58d0d3499831dddfd99dcb3792894
I20260928 20:19:40.052260 6134 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3bc45057888d479a8aa5e19ebe4ad468: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:40.052552 6134 log.cc:824] T 00000000000000000000000000000000 P 3bc45057888d479a8aa5e19ebe4ad468: Log is configured to *not* fsync() on all Append() calls
I20260928 20:19:40.053395 6134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bc45057888d479a8aa5e19ebe4ad468: No bootstrap required, opened a new log
I20260928 20:19:40.055215 6134 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3bc45057888d479a8aa5e19ebe4ad468 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bc45057888d479a8aa5e19ebe4ad468" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36163 } }
I20260928 20:19:40.055392 6134 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3bc45057888d479a8aa5e19ebe4ad468 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:40.055444 6134 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3bc45057888d479a8aa5e19ebe4ad468 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bc45057888d479a8aa5e19ebe4ad468, State: Initialized, Role: FOLLOWER
I20260928 20:19:40.055615 6134 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bc45057888d479a8aa5e19ebe4ad468 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bc45057888d479a8aa5e19ebe4ad468" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36163 } }
I20260928 20:19:40.055744 6134 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3bc45057888d479a8aa5e19ebe4ad468 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:19:40.055822 6134 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3bc45057888d479a8aa5e19ebe4ad468 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:19:40.055876 6134 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3bc45057888d479a8aa5e19ebe4ad468 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:40.056568 6134 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3bc45057888d479a8aa5e19ebe4ad468 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bc45057888d479a8aa5e19ebe4ad468" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36163 } }
I20260928 20:19:40.056699 6134 leader_election.cc:302] T 00000000000000000000000000000000 P 3bc45057888d479a8aa5e19ebe4ad468 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3bc45057888d479a8aa5e19ebe4ad468; no voters:
I20260928 20:19:40.056915 6134 leader_election.cc:288] T 00000000000000000000000000000000 P 3bc45057888d479a8aa5e19ebe4ad468 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:19:40.056967 6139 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3bc45057888d479a8aa5e19ebe4ad468 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:19:40.057178 6139 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3bc45057888d479a8aa5e19ebe4ad468 [term 1 LEADER]: Becoming Leader. State: Replica: 3bc45057888d479a8aa5e19ebe4ad468, State: Running, Role: LEADER
I20260928 20:19:40.057229 6134 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3bc45057888d479a8aa5e19ebe4ad468 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:19:40.057399 6139 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3bc45057888d479a8aa5e19ebe4ad468 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bc45057888d479a8aa5e19ebe4ad468" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36163 } }
I20260928 20:19:40.057794 6139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bc45057888d479a8aa5e19ebe4ad468 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3bc45057888d479a8aa5e19ebe4ad468. Latest consensus state: current_term: 1 leader_uuid: "3bc45057888d479a8aa5e19ebe4ad468" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bc45057888d479a8aa5e19ebe4ad468" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36163 } } }
I20260928 20:19:40.057955 6141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bc45057888d479a8aa5e19ebe4ad468 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3bc45057888d479a8aa5e19ebe4ad468" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bc45057888d479a8aa5e19ebe4ad468" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36163 } } }
I20260928 20:19:40.057964 6139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bc45057888d479a8aa5e19ebe4ad468 [sys.catalog]: This master's current role is: LEADER
I20260928 20:19:40.058056 6141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bc45057888d479a8aa5e19ebe4ad468 [sys.catalog]: This master's current role is: LEADER
I20260928 20:19:40.058529 6145 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:19:40.059159 6145 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 20:19:40 dist-test-slave-b83v krb5kdc[6053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:19:40.061103 6145 catalog_manager.cc:1347] Generated new cluster ID: b03f20fad40d45f69704659488b59baa
I20260928 20:19:40.061157 6145 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:19:40.064131 6138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:40.052741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53880 (local address 127.3.18.62:36163)
0928 20:19:40.052892 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:40.052902 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:40.052944 (+ 42us) server_negotiation.cc:407] Connection header received
0928 20:19:40.053367 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:40.053373 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:40.053420 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:40.053619 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:40.054273 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:40.055142 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:40.056163 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:40.056625 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:40.056750 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:40.059870 (+ 3120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:40.059876 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:40.059876 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:40.059891 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:40.062089 (+ 2198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:40.062561 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:40.062565 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:40.062565 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:40.062676 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:40.062932 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:40.062933 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:40.062934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:40.063249 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:40.063456 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:40.063543 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 20:19:40.063861 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":75,"threads_started":1}
I20260928 20:19:40.065259 3144 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 20:19:40.079392 6145 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:19:40.080325 6145 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:19:40.089614 6145 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3bc45057888d479a8aa5e19ebe4ad468: Generated new TSK 0
I20260928 20:19:40.089875 6145 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 20:19:40.092235 6138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:40.087341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53892 (local address 127.3.18.62:36163)
0928 20:19:40.087397 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:40.087400 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:40.087538 (+ 138us) server_negotiation.cc:407] Connection header received
0928 20:19:40.087630 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:40.087631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:40.087643 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:40.087703 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:40.088202 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:40.089273 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:40.090411 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:40.090682 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:40.090782 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:40.091615 (+ 833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:40.091617 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:40.091617 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:19:40.091914 (+ 297us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:40.091934 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:19:40.091942 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:40.092016 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:40.092095 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 20:19:40.092130 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 20:19:40.095475 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 6066
2026-09-28T20:19:40Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1356 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:19:40 dist-test-slave-b83v krb5kdc[6167](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:19:40 dist-test-slave-b83v krb5kdc[6167](info): set up 2 sockets
Sep 28 20:19:40 dist-test-slave-b83v krb5kdc[6167](info): commencing operation
krb5kdc: starting...
W20260928 20:19:41.168269 3144 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Sep 28 20:19:41 dist-test-slave-b83v krb5kdc[6167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:19:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:19:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.18.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab.
Entry for principal kudu/127.3.18.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab.
I20260928 20:19:41.295544 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:33733
--webserver_interface=127.3.18.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:38687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.18.62:33733
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:19:41.378916 6180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:19:41.379225 6180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:19:41.379273 6180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:19:41.380573 6180 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:19:41.380657 6180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:19:41.380681 6180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:19:41.380703 6180 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:19:41.380733 6180 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:19:41.382347 6180 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:38687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.18.62:33733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:33733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.18.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6180
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:19:41.382671 6180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:19:41.382995 6180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:19:41.388571 6188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:41.388572 6186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:41.388572 6185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:41.389545 6180 server_base.cc:1034] running on GCE node
I20260928 20:19:41.389882 6180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:19:41.390218 6180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:19:41.391536 6180 hybrid_clock.cc:648] HybridClock initialized: now 1790626781391507 us; error 40 us; skew 500 ppm
Sep 28 20:19:41 dist-test-slave-b83v krb5kdc[6167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626781, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:19:41.394820 6180 init.cc:377] Logged in from keytab as kudu/127.3.18.62@KRBTEST.COM (short username kudu)
I20260928 20:19:41.395412 6180 webserver.cc:460] Webserver started at http://127.3.18.62:43179/ using document root <none> and password file <none>
I20260928 20:19:41.395689 6180 fs_manager.cc:359] Metadata directory not provided
I20260928 20:19:41.395767 6180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:19:41.395949 6180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:19:41.397080 6180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/minicluster-data/master-0/data/instance:
uuid: "5bca50243ce740cfa6c50102c2e0b11a"
format_stamp: "Formatted at 2026-09-28 20:19:41 on dist-test-slave-b83v"
server_key: "ac49fcbc85f2a4216d5946ca01d925de"
server_key_iv: "3e08d4d6c5981f963a04f688c9d2274a"
server_key_version: "encryptionkey@0"
I20260928 20:19:41.397457 6180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/minicluster-data/master-0/wal/instance:
uuid: "5bca50243ce740cfa6c50102c2e0b11a"
format_stamp: "Formatted at 2026-09-28 20:19:41 on dist-test-slave-b83v"
server_key: "ac49fcbc85f2a4216d5946ca01d925de"
server_key_iv: "3e08d4d6c5981f963a04f688c9d2274a"
server_key_version: "encryptionkey@0"
I20260928 20:19:41.398972 6180 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 20:19:41.399878 6195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:41.400092 6180 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:19:41.400210 6180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/minicluster-data/master-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/minicluster-data/master-0/wal
uuid: "5bca50243ce740cfa6c50102c2e0b11a"
format_stamp: "Formatted at 2026-09-28 20:19:41 on dist-test-slave-b83v"
server_key: "ac49fcbc85f2a4216d5946ca01d925de"
server_key_iv: "3e08d4d6c5981f963a04f688c9d2274a"
server_key_version: "encryptionkey@0"
I20260928 20:19:41.400292 6180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:19:41.420002 6180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:19:41.422456 6180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:19:41.422645 6180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:19:41.427217 6180 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.62:33733
I20260928 20:19:41.427397 6247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.62:33733 every 8 connection(s)
I20260928 20:19:41.427749 6180 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
I20260928 20:19:41.428880 6248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:41.431224 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 6180
I20260928 20:19:41.431339 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/minicluster-data/master-0/wal/instance
I20260928 20:19:41.431533 3144 external_mini_cluster.cc:1428] Setting key 8663d696afd88e0b47736ce02bf30ff4
I20260928 20:19:41.431995 6248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bca50243ce740cfa6c50102c2e0b11a: Bootstrap starting.
I20260928 20:19:41.432693 6248 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5bca50243ce740cfa6c50102c2e0b11a: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:41.433041 6248 log.cc:824] T 00000000000000000000000000000000 P 5bca50243ce740cfa6c50102c2e0b11a: Log is configured to *not* fsync() on all Append() calls
I20260928 20:19:41.433897 6248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bca50243ce740cfa6c50102c2e0b11a: No bootstrap required, opened a new log
I20260928 20:19:41.436127 6248 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5bca50243ce740cfa6c50102c2e0b11a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bca50243ce740cfa6c50102c2e0b11a" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 33733 } }
I20260928 20:19:41.436296 6248 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5bca50243ce740cfa6c50102c2e0b11a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:41.436319 6248 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5bca50243ce740cfa6c50102c2e0b11a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bca50243ce740cfa6c50102c2e0b11a, State: Initialized, Role: FOLLOWER
I20260928 20:19:41.436468 6248 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5bca50243ce740cfa6c50102c2e0b11a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bca50243ce740cfa6c50102c2e0b11a" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 33733 } }
I20260928 20:19:41.436542 6248 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5bca50243ce740cfa6c50102c2e0b11a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:19:41.436589 6248 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5bca50243ce740cfa6c50102c2e0b11a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:19:41.436657 6248 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5bca50243ce740cfa6c50102c2e0b11a [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:41.437525 6248 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5bca50243ce740cfa6c50102c2e0b11a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bca50243ce740cfa6c50102c2e0b11a" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 33733 } }
I20260928 20:19:41.437649 6248 leader_election.cc:302] T 00000000000000000000000000000000 P 5bca50243ce740cfa6c50102c2e0b11a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5bca50243ce740cfa6c50102c2e0b11a; no voters:
I20260928 20:19:41.437846 6248 leader_election.cc:288] T 00000000000000000000000000000000 P 5bca50243ce740cfa6c50102c2e0b11a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:19:41.437913 6253 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5bca50243ce740cfa6c50102c2e0b11a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:19:41.438066 6253 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5bca50243ce740cfa6c50102c2e0b11a [term 1 LEADER]: Becoming Leader. State: Replica: 5bca50243ce740cfa6c50102c2e0b11a, State: Running, Role: LEADER
I20260928 20:19:41.438154 6253 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5bca50243ce740cfa6c50102c2e0b11a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bca50243ce740cfa6c50102c2e0b11a" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 33733 } }
I20260928 20:19:41.438190 6248 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5bca50243ce740cfa6c50102c2e0b11a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:19:41.438614 6255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bca50243ce740cfa6c50102c2e0b11a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bca50243ce740cfa6c50102c2e0b11a. Latest consensus state: current_term: 1 leader_uuid: "5bca50243ce740cfa6c50102c2e0b11a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bca50243ce740cfa6c50102c2e0b11a" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 33733 } } }
I20260928 20:19:41.438623 6254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bca50243ce740cfa6c50102c2e0b11a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5bca50243ce740cfa6c50102c2e0b11a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bca50243ce740cfa6c50102c2e0b11a" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 33733 } } }
I20260928 20:19:41.438778 6254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bca50243ce740cfa6c50102c2e0b11a [sys.catalog]: This master's current role is: LEADER
I20260928 20:19:41.438984 6255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bca50243ce740cfa6c50102c2e0b11a [sys.catalog]: This master's current role is: LEADER
Sep 28 20:19:41 dist-test-slave-b83v krb5kdc[6167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
W20260928 20:19:41.440199 6268 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5bca50243ce740cfa6c50102c2e0b11a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 20:19:41.440265 6268 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 20:19:41.440326 6262 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:19:41.440510 6262 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:19:41.442268 6262 catalog_manager.cc:1347] Generated new cluster ID: f7f5bd68bb714bdbb8b8a224d69e02cb
I20260928 20:19:41.442332 6262 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:19:41.443804 6251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:41.432198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41476 (local address 127.3.18.62:33733)
0928 20:19:41.432408 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:41.432419 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:41.432446 (+ 27us) server_negotiation.cc:407] Connection header received
0928 20:19:41.432960 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:41.432967 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:41.433021 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:41.433188 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:41.434033 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:41.434918 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:41.436137 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:41.436778 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:41.436873 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:41.439816 (+ 2943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:41.439820 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:41.439820 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:41.439834 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:41.441690 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:41.442090 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:41.442094 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:41.442094 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:41.442215 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:41.442464 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:41.442465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:41.442466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:41.442770 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:41.442920 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:41.443407 (+ 487us) server_negotiation.cc:299] Negotiation successful
0928 20:19:41.443552 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":111,"threads_started":1}
I20260928 20:19:41.462857 6262 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:19:41.463534 6262 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:19:41.474221 6262 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5bca50243ce740cfa6c50102c2e0b11a: Generated new TSK 0
I20260928 20:19:41.474516 6262 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.18.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab.
Entry for principal kudu/127.3.18.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab.
I20260928 20:19:41.511121 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.1:0
--local_ip_for_outbound_sockets=127.3.18.1
--webserver_interface=127.3.18.1
--webserver_port=0
--tserver_master_addrs=127.3.18.62:33733
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab
--principal=kudu/127.3.18.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:38687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:19:41.594743 6274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:19:41.594962 6274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:19:41.594987 6274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:19:41.596302 6274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:19:41.596355 6274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:19:41.596383 6274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.1
I20260928 20:19:41.597852 6274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:38687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab
--principal=kudu/127.3.18.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.18.1
--webserver_port=0
--tserver_master_addrs=127.3.18.62:33733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.1
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:19:41.598124 6274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:19:41.598369 6274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:19:41.604135 6282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:41.604125 6279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:41.604156 6274 server_base.cc:1034] running on GCE node
W20260928 20:19:41.604125 6280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:41.604771 6274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:19:41.605090 6274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:19:41.606349 6274 hybrid_clock.cc:648] HybridClock initialized: now 1790626781606259 us; error 68 us; skew 500 ppm
Sep 28 20:19:41 dist-test-slave-b83v krb5kdc[6167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626781, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:19:41.609279 6274 init.cc:377] Logged in from keytab as kudu/127.3.18.1@KRBTEST.COM (short username kudu)
I20260928 20:19:41.609849 6274 webserver.cc:460] Webserver started at http://127.3.18.1:34035/ using document root <none> and password file <none>
I20260928 20:19:41.610108 6274 fs_manager.cc:359] Metadata directory not provided
I20260928 20:19:41.610179 6274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:19:41.610337 6274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:19:41.611476 6274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-0/data/instance:
uuid: "725b5f8e9342490a9a04084356733b7e"
format_stamp: "Formatted at 2026-09-28 20:19:41 on dist-test-slave-b83v"
server_key: "637930b28101b35bdd92b91c3ba771a0"
server_key_iv: "437049c9fb86d4894fb74a263ca9dd0b"
server_key_version: "encryptionkey@0"
I20260928 20:19:41.611920 6274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-0/wal/instance:
uuid: "725b5f8e9342490a9a04084356733b7e"
format_stamp: "Formatted at 2026-09-28 20:19:41 on dist-test-slave-b83v"
server_key: "637930b28101b35bdd92b91c3ba771a0"
server_key_iv: "437049c9fb86d4894fb74a263ca9dd0b"
server_key_version: "encryptionkey@0"
I20260928 20:19:41.613356 6274 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:19:41.614246 6289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:41.614492 6274 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:19:41.614594 6274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-0/wal
uuid: "725b5f8e9342490a9a04084356733b7e"
format_stamp: "Formatted at 2026-09-28 20:19:41 on dist-test-slave-b83v"
server_key: "637930b28101b35bdd92b91c3ba771a0"
server_key_iv: "437049c9fb86d4894fb74a263ca9dd0b"
server_key_version: "encryptionkey@0"
I20260928 20:19:41.614683 6274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:19:41.641820 6274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:19:41.644477 6274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:19:41.644656 6274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:19:41.644932 6274 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:19:41.645272 6274 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:19:41.645401 6274 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:41.645447 6274 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:19:41.645495 6274 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:41.651548 6274 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.1:42197
I20260928 20:19:41.651597 6402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.1:42197 every 8 connection(s)
I20260928 20:19:41.651994 6274 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
I20260928 20:19:41.656653 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 6274
I20260928 20:19:41.656800 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-0/wal/instance
I20260928 20:19:41.657028 3144 external_mini_cluster.cc:1428] Setting key 49531a98ab2b9971f7b89336118d5b8a
Sep 28 20:19:41 dist-test-slave-b83v krb5kdc[6167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626781, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.1@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:19:41.663620 6251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:41.652796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:47357 (local address 127.3.18.62:33733)
0928 20:19:41.652872 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:41.652875 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:41.653112 (+ 237us) server_negotiation.cc:407] Connection header received
0928 20:19:41.653707 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:41.653708 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:41.653719 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:41.653769 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:41.654989 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:41.655608 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:41.656828 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:41.657122 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:41.657197 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:41.659864 (+ 2667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:41.659866 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:41.659867 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:41.659876 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:41.661873 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:41.662453 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:41.662455 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:41.662456 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:41.662528 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:41.662820 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:41.662821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:41.662822 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:41.662995 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:41.663074 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:41.663360 (+ 286us) server_negotiation.cc:299] Negotiation successful
0928 20:19:41.663473 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 20:19:41.663717 6405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:41.652756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:33733 (local address 127.3.18.1:47357)
0928 20:19:41.652931 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:41.652974 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:41.653608 (+ 634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:41.653845 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:41.653849 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:41.654465 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:41.654669 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:41.654675 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:41.655753 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:41.655754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:41.656679 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:41.656682 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:41.657221 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:41.657223 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:41.657426 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:41.658060 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:19:41.658069 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:41.659656 (+ 1587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:41.661989 (+ 2333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:41.662000 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:41.662003 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:41.662285 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:41.662608 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:41.662608 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:41.662608 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:41.662748 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:41.663066 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:41.663071 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:41.663221 (+ 150us) client_negotiation.cc:769] Sending connection context
0928 20:19:41.663308 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 20:19:41.663494 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":100,"threads_started":1}
I20260928 20:19:41.664315 6403 heartbeater.cc:347] Connected to a master server at 127.3.18.62:33733
I20260928 20:19:41.664467 6403 heartbeater.cc:464] Registering TS with master...
I20260928 20:19:41.664706 6403 heartbeater.cc:511] Master 127.3.18.62:33733 requested a full tablet report, sending...
I20260928 20:19:41.665295 6212 ts_manager.cc:194] Registered new tserver with Master: 725b5f8e9342490a9a04084356733b7e (127.3.18.1:42197)
I20260928 20:19:41.665977 6212 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.18.1@KRBTEST.COM'} at 127.3.18.1:47357
WARNING: no policy specified for kudu/127.3.18.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab.
Entry for principal kudu/127.3.18.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab.
I20260928 20:19:41.687278 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.2:0
--local_ip_for_outbound_sockets=127.3.18.2
--webserver_interface=127.3.18.2
--webserver_port=0
--tserver_master_addrs=127.3.18.62:33733
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab
--principal=kudu/127.3.18.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:38687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:19:41.771646 6408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:19:41.771935 6408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:19:41.771979 6408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:19:41.773160 6408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:19:41.773238 6408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:19:41.773284 6408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.2
I20260928 20:19:41.774739 6408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:38687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab
--principal=kudu/127.3.18.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.18.2
--webserver_port=0
--tserver_master_addrs=127.3.18.62:33733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.2
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:19:41.775032 6408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:19:41.775275 6408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:19:41.780555 6413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:41.780582 6416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:41.780546 6414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:41.780638 6408 server_base.cc:1034] running on GCE node
I20260928 20:19:41.781070 6408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:19:41.781339 6408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:19:41.782524 6408 hybrid_clock.cc:648] HybridClock initialized: now 1790626781782505 us; error 43 us; skew 500 ppm
Sep 28 20:19:41 dist-test-slave-b83v krb5kdc[6167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626781, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:19:41.785056 6408 init.cc:377] Logged in from keytab as kudu/127.3.18.2@KRBTEST.COM (short username kudu)
I20260928 20:19:41.785522 6408 webserver.cc:460] Webserver started at http://127.3.18.2:45995/ using document root <none> and password file <none>
I20260928 20:19:41.785751 6408 fs_manager.cc:359] Metadata directory not provided
I20260928 20:19:41.785794 6408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:19:41.785936 6408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:19:41.787017 6408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-1/data/instance:
uuid: "2fa5d8503642432e92c4c9fe71f39041"
format_stamp: "Formatted at 2026-09-28 20:19:41 on dist-test-slave-b83v"
server_key: "36d6119baa6ce6b033f86e8c11ce8182"
server_key_iv: "2cf76e32bc360af607f385ed7e53ef91"
server_key_version: "encryptionkey@0"
I20260928 20:19:41.787385 6408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-1/wal/instance:
uuid: "2fa5d8503642432e92c4c9fe71f39041"
format_stamp: "Formatted at 2026-09-28 20:19:41 on dist-test-slave-b83v"
server_key: "36d6119baa6ce6b033f86e8c11ce8182"
server_key_iv: "2cf76e32bc360af607f385ed7e53ef91"
server_key_version: "encryptionkey@0"
I20260928 20:19:41.788828 6408 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 20:19:41.789719 6423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:41.790016 6408 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 20:19:41.790179 6408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-1/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-1/wal
uuid: "2fa5d8503642432e92c4c9fe71f39041"
format_stamp: "Formatted at 2026-09-28 20:19:41 on dist-test-slave-b83v"
server_key: "36d6119baa6ce6b033f86e8c11ce8182"
server_key_iv: "2cf76e32bc360af607f385ed7e53ef91"
server_key_version: "encryptionkey@0"
I20260928 20:19:41.790272 6408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:19:41.798919 6408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:19:41.801136 6408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:19:41.801288 6408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:19:41.801522 6408 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:19:41.801818 6408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:19:41.801870 6408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:41.801910 6408 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:19:41.801939 6408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:41.808130 6408 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.2:37633
I20260928 20:19:41.808168 6536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.2:37633 every 8 connection(s)
I20260928 20:19:41.808565 6408 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
I20260928 20:19:41.812770 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 6408
I20260928 20:19:41.812920 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-1/wal/instance
I20260928 20:19:41.813128 3144 external_mini_cluster.cc:1428] Setting key 1cfc3bb18046cc9a19d244a63be4aba8
Sep 28 20:19:41 dist-test-slave-b83v krb5kdc[6167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626781, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.2@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:19:41.818987 6251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:41.809392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:42121 (local address 127.3.18.62:33733)
0928 20:19:41.809463 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:41.809465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:41.809655 (+ 190us) server_negotiation.cc:407] Connection header received
0928 20:19:41.810182 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:41.810183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:41.810196 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:41.810246 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:41.811232 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:41.811745 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:41.812865 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:41.813151 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:41.813238 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:41.815699 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:41.815701 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:41.815701 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:41.815711 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:41.817260 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:41.817845 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:41.817846 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:41.817846 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:41.817897 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:41.818218 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:41.818220 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:41.818220 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:41.818356 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:41.818431 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:41.818851 (+ 420us) server_negotiation.cc:299] Negotiation successful
0928 20:19:41.818920 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 20:19:41.819258 6539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:41.809365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:33733 (local address 127.3.18.2:42121)
0928 20:19:41.809525 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:41.809567 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:41.810051 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:41.810316 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:41.810326 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:41.810916 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:41.811110 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:41.811115 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:41.811915 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:41.811917 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:41.812739 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:41.812742 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:41.813249 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:41.813250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:41.813390 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:41.813953 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:19:41.813961 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:41.815601 (+ 1640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:41.817360 (+ 1759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:41.817367 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:41.817370 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:41.817755 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:41.817965 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:41.817966 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:41.817966 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:41.818120 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:41.818442 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:41.818449 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:41.818635 (+ 186us) client_negotiation.cc:769] Sending connection context
0928 20:19:41.818763 (+ 128us) client_negotiation.cc:240] Negotiation successful
0928 20:19:41.818960 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":89,"threads_started":1}
I20260928 20:19:41.819887 6537 heartbeater.cc:347] Connected to a master server at 127.3.18.62:33733
I20260928 20:19:41.819978 6537 heartbeater.cc:464] Registering TS with master...
I20260928 20:19:41.820166 6537 heartbeater.cc:511] Master 127.3.18.62:33733 requested a full tablet report, sending...
I20260928 20:19:41.820537 6212 ts_manager.cc:194] Registered new tserver with Master: 2fa5d8503642432e92c4c9fe71f39041 (127.3.18.2:37633)
I20260928 20:19:41.821103 6212 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.18.2@KRBTEST.COM'} at 127.3.18.2:42121
WARNING: no policy specified for kudu/127.3.18.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab.
Entry for principal kudu/127.3.18.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab.
I20260928 20:19:41.842464 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.3:0
--local_ip_for_outbound_sockets=127.3.18.3
--webserver_interface=127.3.18.3
--webserver_port=0
--tserver_master_addrs=127.3.18.62:33733
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab
--principal=kudu/127.3.18.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:38687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:19:41.928385 6542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:19:41.928705 6542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:19:41.928750 6542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:19:41.930171 6542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:19:41.930275 6542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:19:41.930327 6542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.3
I20260928 20:19:41.932516 6542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:38687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab
--principal=kudu/127.3.18.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.18.3
--webserver_port=0
--tserver_master_addrs=127.3.18.62:33733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.3
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:19:41.932865 6542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:19:41.933140 6542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:19:41.939386 6547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:41.939441 6548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:41.939929 6550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:41.939970 6542 server_base.cc:1034] running on GCE node
I20260928 20:19:41.940222 6542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:19:41.940503 6542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:19:41.941730 6542 hybrid_clock.cc:648] HybridClock initialized: now 1790626781941694 us; error 41 us; skew 500 ppm
Sep 28 20:19:41 dist-test-slave-b83v krb5kdc[6167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626781, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:19:41.964633 6542 init.cc:377] Logged in from keytab as kudu/127.3.18.3@KRBTEST.COM (short username kudu)
I20260928 20:19:41.965234 6542 webserver.cc:460] Webserver started at http://127.3.18.3:37417/ using document root <none> and password file <none>
I20260928 20:19:41.965493 6542 fs_manager.cc:359] Metadata directory not provided
I20260928 20:19:41.965565 6542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:19:41.965726 6542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:19:41.966971 6542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-2/data/instance:
uuid: "68102ec8a8b54051b202ce21c7e63c07"
format_stamp: "Formatted at 2026-09-28 20:19:41 on dist-test-slave-b83v"
server_key: "5315c9ad42f2bd06a806f64f8fe3e251"
server_key_iv: "30793d5919e2c24ce54aca1768bf4db2"
server_key_version: "encryptionkey@0"
I20260928 20:19:41.967360 6542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-2/wal/instance:
uuid: "68102ec8a8b54051b202ce21c7e63c07"
format_stamp: "Formatted at 2026-09-28 20:19:41 on dist-test-slave-b83v"
server_key: "5315c9ad42f2bd06a806f64f8fe3e251"
server_key_iv: "30793d5919e2c24ce54aca1768bf4db2"
server_key_version: "encryptionkey@0"
I20260928 20:19:41.968863 6542 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 20:19:41.969825 6557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:41.969985 6542 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:19:41.970110 6542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-2/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-2/wal
uuid: "68102ec8a8b54051b202ce21c7e63c07"
format_stamp: "Formatted at 2026-09-28 20:19:41 on dist-test-slave-b83v"
server_key: "5315c9ad42f2bd06a806f64f8fe3e251"
server_key_iv: "30793d5919e2c24ce54aca1768bf4db2"
server_key_version: "encryptionkey@0"
I20260928 20:19:41.970192 6542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:19:41.986948 6542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:19:41.989440 6542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:19:41.989605 6542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:19:41.989856 6542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:19:41.990162 6542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:19:41.990214 6542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:41.990257 6542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:19:41.990283 6542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:41.996658 6542 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.3:34405
I20260928 20:19:41.996840 6670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.3:34405 every 8 connection(s)
I20260928 20:19:41.997087 6542 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
I20260928 20:19:41.998512 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 6542
I20260928 20:19:41.998592 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790626764250411-3144-0/minicluster-data/ts-2/wal/instance
I20260928 20:19:41.998768 3144 external_mini_cluster.cc:1428] Setting key 793fe38768d8972c822cdc65a5c9c87b
Sep 28 20:19:41 dist-test-slave-b83v krb5kdc[6167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626781, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.3@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:19:42.007174 6251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:41.997941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:53777 (local address 127.3.18.62:33733)
0928 20:19:41.998025 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:41.998027 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:41.998199 (+ 172us) server_negotiation.cc:407] Connection header received
0928 20:19:41.998689 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:41.998691 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:41.998705 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:41.998753 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:41.999983 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:42.000499 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:42.001542 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:42.001746 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:42.001831 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:42.004251 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:42.004253 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:42.004253 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:42.004261 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:42.005720 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:42.006165 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:42.006167 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:42.006167 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:42.006229 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:42.006484 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:42.006485 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:42.006486 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:42.006608 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:42.006684 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:42.006986 (+ 302us) server_negotiation.cc:299] Negotiation successful
0928 20:19:42.007070 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 20:19:42.007416 6673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:41.997879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:33733 (local address 127.3.18.3:53777)
0928 20:19:41.998037 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:41.998079 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:41.998588 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:41.998830 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:41.998834 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:41.999566 (+ 732us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:41.999860 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:41.999866 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:42.000606 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:42.000608 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:42.001454 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:42.001456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:42.001848 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:42.001849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:42.001980 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:42.002622 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:19:42.002631 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:42.004137 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:42.005829 (+ 1692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:42.005835 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:42.005838 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:42.006058 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:42.006296 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:42.006297 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:42.006297 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:42.006416 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:42.006675 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:42.006681 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:42.006840 (+ 159us) client_negotiation.cc:769] Sending connection context
0928 20:19:42.006965 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 20:19:42.007146 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":91,"threads_started":1}
I20260928 20:19:42.008088 6671 heartbeater.cc:347] Connected to a master server at 127.3.18.62:33733
I20260928 20:19:42.008405 6671 heartbeater.cc:464] Registering TS with master...
I20260928 20:19:42.008666 6671 heartbeater.cc:511] Master 127.3.18.62:33733 requested a full tablet report, sending...
I20260928 20:19:42.009132 6212 ts_manager.cc:194] Registered new tserver with Master: 68102ec8a8b54051b202ce21c7e63c07 (127.3.18.3:34405)
I20260928 20:19:42.009542 6212 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.18.3@KRBTEST.COM'} at 127.3.18.3:53777
I20260928 20:19:42.010675 3144 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 20:19:42 dist-test-slave-b83v krb5kdc[6167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626782, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 20:19:42 dist-test-slave-b83v krb5kdc[6167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626782, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:19:42.037171 6251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:42.027523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41490 (local address 127.3.18.62:33733)
0928 20:19:42.027710 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:42.027713 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:42.027909 (+ 196us) server_negotiation.cc:407] Connection header received
0928 20:19:42.028017 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:42.028019 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:42.028044 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:42.028110 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:42.028759 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:42.029291 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:42.030092 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:42.030299 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:42.030379 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:42.033431 (+ 3052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:42.033433 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:42.033433 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:42.033442 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:42.035572 (+ 2130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:42.036199 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:42.036200 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:42.036200 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:42.036273 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:42.036496 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:42.036496 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:42.036497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:42.036675 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:42.036765 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:42.037006 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 20:19:42.037093 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153}
W20260928 20:19:42.037513 6212 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:41490
I20260928 20:19:42.038094 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 6274
I20260928 20:19:42.043015 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 6408
I20260928 20:19:42.048097 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 6542
I20260928 20:19:42.053200 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 6180
2026-09-28T20:19:42Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1960 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:19:42 dist-test-slave-b83v krb5kdc[6683](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:19:42 dist-test-slave-b83v krb5kdc[6683](info): set up 2 sockets
Sep 28 20:19:42 dist-test-slave-b83v krb5kdc[6683](info): commencing operation
krb5kdc: starting...
W20260928 20:19:43.096032 3144 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Sep 28 20:19:43 dist-test-slave-b83v krb5kdc[6683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:19:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:19:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.18.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab.
Entry for principal kudu/127.3.18.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab.
I20260928 20:19:43.226974 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:44021
--webserver_interface=127.3.18.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:35033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.18.62:44021
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:19:43.313817 6696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:19:43.314070 6696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:19:43.314108 6696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:19:43.315301 6696 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:19:43.315374 6696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:19:43.315398 6696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:19:43.315429 6696 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:19:43.315476 6696 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:19:43.317066 6696 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:35033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.18.62:44021
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:44021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.18.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:19:43.317363 6696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:19:43.317597 6696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:19:43.322695 6704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:43.322719 6701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:43.323222 6702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:43.323423 6696 server_base.cc:1034] running on GCE node
I20260928 20:19:43.323707 6696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:19:43.324092 6696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:19:43.325248 6696 hybrid_clock.cc:648] HybridClock initialized: now 1790626783325216 us; error 43 us; skew 500 ppm
Sep 28 20:19:43 dist-test-slave-b83v krb5kdc[6683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626783, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:19:43.328042 6696 init.cc:377] Logged in from keytab as kudu/127.3.18.62@KRBTEST.COM (short username kudu)
I20260928 20:19:43.328565 6696 webserver.cc:460] Webserver started at http://127.3.18.62:44421/ using document root <none> and password file <none>
I20260928 20:19:43.328820 6696 fs_manager.cc:359] Metadata directory not provided
I20260928 20:19:43.328902 6696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:19:43.329051 6696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:19:43.330159 6696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/minicluster-data/master-0/data/instance:
uuid: "cbb310815b004bd4be687309995ddbbe"
format_stamp: "Formatted at 2026-09-28 20:19:43 on dist-test-slave-b83v"
server_key: "b7371c34f4279f0382e8051d0177dc90"
server_key_iv: "7cc5d4cc654bde0ee9de11a7d65c921c"
server_key_version: "encryptionkey@0"
I20260928 20:19:43.330575 6696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/minicluster-data/master-0/wal/instance:
uuid: "cbb310815b004bd4be687309995ddbbe"
format_stamp: "Formatted at 2026-09-28 20:19:43 on dist-test-slave-b83v"
server_key: "b7371c34f4279f0382e8051d0177dc90"
server_key_iv: "7cc5d4cc654bde0ee9de11a7d65c921c"
server_key_version: "encryptionkey@0"
I20260928 20:19:43.332127 6696 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 20:19:43.333084 6711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:43.333288 6696 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:19:43.333407 6696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/minicluster-data/master-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/minicluster-data/master-0/wal
uuid: "cbb310815b004bd4be687309995ddbbe"
format_stamp: "Formatted at 2026-09-28 20:19:43 on dist-test-slave-b83v"
server_key: "b7371c34f4279f0382e8051d0177dc90"
server_key_iv: "7cc5d4cc654bde0ee9de11a7d65c921c"
server_key_version: "encryptionkey@0"
I20260928 20:19:43.333516 6696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:19:43.354223 6696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:19:43.356657 6696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:19:43.356856 6696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:19:43.361058 6696 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.62:44021
I20260928 20:19:43.361112 6763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.62:44021 every 8 connection(s)
I20260928 20:19:43.361675 6696 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
I20260928 20:19:43.362459 6764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:43.362905 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 6696
I20260928 20:19:43.362996 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/minicluster-data/master-0/wal/instance
I20260928 20:19:43.363188 3144 external_mini_cluster.cc:1428] Setting key 9d1d361ede0db529a8c22f372b5df6ba
I20260928 20:19:43.365760 6764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbb310815b004bd4be687309995ddbbe: Bootstrap starting.
I20260928 20:19:43.366425 6764 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbb310815b004bd4be687309995ddbbe: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:43.366850 6764 log.cc:824] T 00000000000000000000000000000000 P cbb310815b004bd4be687309995ddbbe: Log is configured to *not* fsync() on all Append() calls
I20260928 20:19:43.367853 6764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbb310815b004bd4be687309995ddbbe: No bootstrap required, opened a new log
I20260928 20:19:43.370616 6764 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbb310815b004bd4be687309995ddbbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbb310815b004bd4be687309995ddbbe" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 44021 } }
I20260928 20:19:43.370795 6764 raft_consensus.cc:374] T 00000000000000000000000000000000 P cbb310815b004bd4be687309995ddbbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:43.370819 6764 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbb310815b004bd4be687309995ddbbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbb310815b004bd4be687309995ddbbe, State: Initialized, Role: FOLLOWER
Sep 28 20:19:43 dist-test-slave-b83v krb5kdc[6683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:19:43.370960 6764 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbb310815b004bd4be687309995ddbbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbb310815b004bd4be687309995ddbbe" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 44021 } }
I20260928 20:19:43.371028 6764 raft_consensus.cc:388] T 00000000000000000000000000000000 P cbb310815b004bd4be687309995ddbbe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:19:43.371086 6764 raft_consensus.cc:482] T 00000000000000000000000000000000 P cbb310815b004bd4be687309995ddbbe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:19:43.371134 6764 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbb310815b004bd4be687309995ddbbe [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:43.371743 6764 raft_consensus.cc:504] T 00000000000000000000000000000000 P cbb310815b004bd4be687309995ddbbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbb310815b004bd4be687309995ddbbe" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 44021 } }
I20260928 20:19:43.371912 6764 leader_election.cc:302] T 00000000000000000000000000000000 P cbb310815b004bd4be687309995ddbbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cbb310815b004bd4be687309995ddbbe; no voters:
I20260928 20:19:43.372100 6764 leader_election.cc:288] T 00000000000000000000000000000000 P cbb310815b004bd4be687309995ddbbe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:19:43.372172 6769 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cbb310815b004bd4be687309995ddbbe [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:19:43.372396 6764 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbb310815b004bd4be687309995ddbbe [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:19:43.372396 6769 raft_consensus.cc:686] T 00000000000000000000000000000000 P cbb310815b004bd4be687309995ddbbe [term 1 LEADER]: Becoming Leader. State: Replica: cbb310815b004bd4be687309995ddbbe, State: Running, Role: LEADER
I20260928 20:19:43.372560 6769 consensus_queue.cc:237] T 00000000000000000000000000000000 P cbb310815b004bd4be687309995ddbbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbb310815b004bd4be687309995ddbbe" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 44021 } }
I20260928 20:19:43.372953 6770 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbb310815b004bd4be687309995ddbbe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cbb310815b004bd4be687309995ddbbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbb310815b004bd4be687309995ddbbe" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 44021 } } }
I20260928 20:19:43.373060 6770 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbb310815b004bd4be687309995ddbbe [sys.catalog]: This master's current role is: LEADER
I20260928 20:19:43.372969 6771 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbb310815b004bd4be687309995ddbbe [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbb310815b004bd4be687309995ddbbe. Latest consensus state: current_term: 1 leader_uuid: "cbb310815b004bd4be687309995ddbbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbb310815b004bd4be687309995ddbbe" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 44021 } } }
I20260928 20:19:43.373490 6771 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbb310815b004bd4be687309995ddbbe [sys.catalog]: This master's current role is: LEADER
I20260928 20:19:43.373620 6778 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:19:43.374346 6778 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:19:43.375835 6767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:43.363907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47662 (local address 127.3.18.62:44021)
0928 20:19:43.364106 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:43.364115 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:43.364154 (+ 39us) server_negotiation.cc:407] Connection header received
0928 20:19:43.364602 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:43.364609 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:43.364664 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:43.364815 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:43.365640 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:43.366808 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:43.367924 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:43.368501 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:43.368613 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:43.371551 (+ 2938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:43.371563 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:43.371563 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:43.371576 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:43.373933 (+ 2357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:43.374402 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:43.374406 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:43.374406 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:43.374540 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:43.374782 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:43.374783 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:43.374783 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:43.375148 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:43.375304 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:43.375447 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 20:19:43.375592 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":95,"threads_started":1}
I20260928 20:19:43.376273 6778 catalog_manager.cc:1347] Generated new cluster ID: 6879eb7b2a0b4b349dd3a0aea65cbb08
I20260928 20:19:43.376322 6778 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:19:43.384361 6778 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:19:43.384938 6778 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:19:43.389950 6778 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cbb310815b004bd4be687309995ddbbe: Generated new TSK 0
I20260928 20:19:43.390139 6778 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.18.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab.
Entry for principal kudu/127.3.18.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab.
I20260928 20:19:43.422407 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.1:0
--local_ip_for_outbound_sockets=127.3.18.1
--webserver_interface=127.3.18.1
--webserver_port=0
--tserver_master_addrs=127.3.18.62:44021
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab
--principal=kudu/127.3.18.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:35033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:19:43.508710 6790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:19:43.508945 6790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:19:43.508985 6790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:19:43.510222 6790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:19:43.510273 6790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:19:43.510303 6790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.1
I20260928 20:19:43.511956 6790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:35033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab
--principal=kudu/127.3.18.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.18.1
--webserver_port=0
--tserver_master_addrs=127.3.18.62:44021
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.1
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:19:43.512219 6790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:19:43.512477 6790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:19:43.517993 6795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:43.518088 6796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:43.518172 6790 server_base.cc:1034] running on GCE node
W20260928 20:19:43.518088 6798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:43.518590 6790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:19:43.518842 6790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:19:43.520040 6790 hybrid_clock.cc:648] HybridClock initialized: now 1790626783520024 us; error 33 us; skew 500 ppm
Sep 28 20:19:43 dist-test-slave-b83v krb5kdc[6683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626783, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:19:43.522658 6790 init.cc:377] Logged in from keytab as kudu/127.3.18.1@KRBTEST.COM (short username kudu)
I20260928 20:19:43.523222 6790 webserver.cc:460] Webserver started at http://127.3.18.1:37769/ using document root <none> and password file <none>
I20260928 20:19:43.523474 6790 fs_manager.cc:359] Metadata directory not provided
I20260928 20:19:43.523548 6790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:19:43.523701 6790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:19:43.524888 6790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/minicluster-data/ts-0/data/instance:
uuid: "d2b37ba78910498bbc61c017d84c3fcc"
format_stamp: "Formatted at 2026-09-28 20:19:43 on dist-test-slave-b83v"
server_key: "5da44baa30a66e5ae359ea18fdadf5d8"
server_key_iv: "2ea3193b0423afcc8246ac50d7d3de72"
server_key_version: "encryptionkey@0"
I20260928 20:19:43.525282 6790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/minicluster-data/ts-0/wal/instance:
uuid: "d2b37ba78910498bbc61c017d84c3fcc"
format_stamp: "Formatted at 2026-09-28 20:19:43 on dist-test-slave-b83v"
server_key: "5da44baa30a66e5ae359ea18fdadf5d8"
server_key_iv: "2ea3193b0423afcc8246ac50d7d3de72"
server_key_version: "encryptionkey@0"
I20260928 20:19:43.526798 6790 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:19:43.527725 6805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:43.528051 6790 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:19:43.528160 6790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/minicluster-data/ts-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/minicluster-data/ts-0/wal
uuid: "d2b37ba78910498bbc61c017d84c3fcc"
format_stamp: "Formatted at 2026-09-28 20:19:43 on dist-test-slave-b83v"
server_key: "5da44baa30a66e5ae359ea18fdadf5d8"
server_key_iv: "2ea3193b0423afcc8246ac50d7d3de72"
server_key_version: "encryptionkey@0"
I20260928 20:19:43.528254 6790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:19:43.543915 6790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:19:43.546103 6790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:19:43.546293 6790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:19:43.546559 6790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:19:43.546937 6790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:19:43.547001 6790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:43.547040 6790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:19:43.547091 6790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:43.553669 6790 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.1:46105
I20260928 20:19:43.553711 6918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.1:46105 every 8 connection(s)
I20260928 20:19:43.554078 6790 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
I20260928 20:19:43.557648 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 6790
I20260928 20:19:43.557796 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/minicluster-data/ts-0/wal/instance
I20260928 20:19:43.558007 3144 external_mini_cluster.cc:1428] Setting key 778e61801a8c4470c973c032d787dff2
Sep 28 20:19:43 dist-test-slave-b83v krb5kdc[6683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626783, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.1@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:19:43.565258 6767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:43.555036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:41883 (local address 127.3.18.62:44021)
0928 20:19:43.555131 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:43.555134 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:43.555329 (+ 195us) server_negotiation.cc:407] Connection header received
0928 20:19:43.556411 (+ 1082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:43.556413 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:43.556427 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:43.556500 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:43.557204 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:43.557761 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:43.559181 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:43.559480 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:43.559562 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:43.562226 (+ 2664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:43.562228 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:43.562229 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:43.562236 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:43.563864 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:43.564266 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:43.564268 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:43.564268 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:43.564334 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:43.564577 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:43.564578 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:43.564578 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:43.564693 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:43.564766 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:43.565071 (+ 305us) server_negotiation.cc:299] Negotiation successful
0928 20:19:43.565163 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 20:19:43.565466 6921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:43.555004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:44021 (local address 127.3.18.1:41883)
0928 20:19:43.555195 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:43.555234 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:43.556282 (+ 1048us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:43.556583 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:43.556587 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:43.556874 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:43.557097 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:43.557101 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:43.557888 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:43.557889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:43.559095 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:43.559099 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:43.559569 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:43.559570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:43.559726 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:43.560395 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:19:43.560404 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:43.562114 (+ 1710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:43.563953 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:43.563961 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:43.563963 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:43.564187 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:43.564400 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:43.564401 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:43.564402 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:43.564510 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:43.564757 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:43.564764 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:43.564938 (+ 174us) client_negotiation.cc:769] Sending connection context
0928 20:19:43.565051 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 20:19:43.565215 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":90,"threads_started":1}
I20260928 20:19:43.566020 6919 heartbeater.cc:347] Connected to a master server at 127.3.18.62:44021
I20260928 20:19:43.566107 6919 heartbeater.cc:464] Registering TS with master...
I20260928 20:19:43.566304 6919 heartbeater.cc:511] Master 127.3.18.62:44021 requested a full tablet report, sending...
I20260928 20:19:43.567001 6728 ts_manager.cc:194] Registered new tserver with Master: d2b37ba78910498bbc61c017d84c3fcc (127.3.18.1:46105)
WARNING: no policy specified for kudu/127.3.18.2@KRBTEST.COM; defaulting to no policy
I20260928 20:19:43.567970 6728 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.18.1@KRBTEST.COM'} at 127.3.18.1:41883
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab.
Entry for principal kudu/127.3.18.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab.
I20260928 20:19:43.587672 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.2:0
--local_ip_for_outbound_sockets=127.3.18.2
--webserver_interface=127.3.18.2
--webserver_port=0
--tserver_master_addrs=127.3.18.62:44021
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab
--principal=kudu/127.3.18.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:35033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:19:43.672511 6924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:19:43.672793 6924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:19:43.672821 6924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:19:43.674206 6924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:19:43.674264 6924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:19:43.674294 6924 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.2
I20260928 20:19:43.676050 6924 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:35033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab
--principal=kudu/127.3.18.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.18.2
--webserver_port=0
--tserver_master_addrs=127.3.18.62:44021
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6924
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.2
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:19:43.676328 6924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:19:43.676592 6924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:19:43.681951 6930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:43.681954 6932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:43.681952 6929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:43.681978 6924 server_base.cc:1034] running on GCE node
I20260928 20:19:43.682359 6924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:19:43.682629 6924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:19:43.683842 6924 hybrid_clock.cc:648] HybridClock initialized: now 1790626783683818 us; error 37 us; skew 500 ppm
Sep 28 20:19:43 dist-test-slave-b83v krb5kdc[6683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626783, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:19:43.686571 6924 init.cc:377] Logged in from keytab as kudu/127.3.18.2@KRBTEST.COM (short username kudu)
I20260928 20:19:43.687104 6924 webserver.cc:460] Webserver started at http://127.3.18.2:43421/ using document root <none> and password file <none>
I20260928 20:19:43.687342 6924 fs_manager.cc:359] Metadata directory not provided
I20260928 20:19:43.687417 6924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:19:43.687579 6924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:19:43.688755 6924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/minicluster-data/ts-1/data/instance:
uuid: "34c7edf1481e4485b7bf4336fb284b21"
format_stamp: "Formatted at 2026-09-28 20:19:43 on dist-test-slave-b83v"
server_key: "1f80764b23d91cf87d5f64128f0d919b"
server_key_iv: "e1b0064b671945c5f493ec6dbaa085f7"
server_key_version: "encryptionkey@0"
I20260928 20:19:43.689179 6924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/minicluster-data/ts-1/wal/instance:
uuid: "34c7edf1481e4485b7bf4336fb284b21"
format_stamp: "Formatted at 2026-09-28 20:19:43 on dist-test-slave-b83v"
server_key: "1f80764b23d91cf87d5f64128f0d919b"
server_key_iv: "e1b0064b671945c5f493ec6dbaa085f7"
server_key_version: "encryptionkey@0"
I20260928 20:19:43.690604 6924 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:19:43.691406 6939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:43.691625 6924 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:19:43.691728 6924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/minicluster-data/ts-1/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/minicluster-data/ts-1/wal
uuid: "34c7edf1481e4485b7bf4336fb284b21"
format_stamp: "Formatted at 2026-09-28 20:19:43 on dist-test-slave-b83v"
server_key: "1f80764b23d91cf87d5f64128f0d919b"
server_key_iv: "e1b0064b671945c5f493ec6dbaa085f7"
server_key_version: "encryptionkey@0"
I20260928 20:19:43.691850 6924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:19:43.701623 6924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:19:43.703704 6924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:19:43.703917 6924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:19:43.704167 6924 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:19:43.704504 6924 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:19:43.704559 6924 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:43.704602 6924 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:19:43.704630 6924 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:43.710803 6924 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.2:39537
I20260928 20:19:43.710847 7052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.2:39537 every 8 connection(s)
I20260928 20:19:43.711221 6924 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
I20260928 20:19:43.713383 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 6924
I20260928 20:19:43.713498 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/minicluster-data/ts-1/wal/instance
I20260928 20:19:43.713701 3144 external_mini_cluster.cc:1428] Setting key 35aa5c6109f336d257754e38a527bbb1
Sep 28 20:19:43 dist-test-slave-b83v krb5kdc[6683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626783, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.2@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.18.3@KRBTEST.COM; defaulting to no policy
I20260928 20:19:43.723114 6767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:43.712086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:39879 (local address 127.3.18.62:44021)
0928 20:19:43.712132 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:43.712134 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:43.712370 (+ 236us) server_negotiation.cc:407] Connection header received
0928 20:19:43.713413 (+ 1043us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:43.713414 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:43.713425 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:43.713535 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:43.714422 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:43.714932 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:43.716327 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:43.716661 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:43.716768 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:43.719327 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:43.719329 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:43.719329 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:43.719337 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:43.721608 (+ 2271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:43.722058 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:43.722060 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:43.722060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:43.722128 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:43.722403 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:43.722404 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:43.722405 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:43.722538 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:43.722620 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:43.722928 (+ 308us) server_negotiation.cc:299] Negotiation successful
0928 20:19:43.723016 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 20:19:43.723327 7055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:43.712063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:44021 (local address 127.3.18.2:39879)
0928 20:19:43.712234 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:43.712271 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:43.713270 (+ 999us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:43.713619 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:43.713624 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:43.714034 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:43.714327 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:43.714334 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:43.715034 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:43.715035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:43.716235 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:43.716238 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:43.716764 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:43.716765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:43.716902 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:43.717489 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:19:43.717497 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:43.719217 (+ 1720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:43.721710 (+ 2493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:43.721717 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:43.721719 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:43.721964 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:43.722203 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:43.722204 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:43.722204 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:43.722328 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:43.722609 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:43.722616 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:43.722774 (+ 158us) client_negotiation.cc:769] Sending connection context
0928 20:19:43.722904 (+ 130us) client_negotiation.cc:240] Negotiation successful
0928 20:19:43.723075 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":114,"threads_started":1}
I20260928 20:19:43.723908 7053 heartbeater.cc:347] Connected to a master server at 127.3.18.62:44021
I20260928 20:19:43.724040 7053 heartbeater.cc:464] Registering TS with master...
I20260928 20:19:43.724341 7053 heartbeater.cc:511] Master 127.3.18.62:44021 requested a full tablet report, sending...
I20260928 20:19:43.724737 6728 ts_manager.cc:194] Registered new tserver with Master: 34c7edf1481e4485b7bf4336fb284b21 (127.3.18.2:39537)
I20260928 20:19:43.725204 6728 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.18.2@KRBTEST.COM'} at 127.3.18.2:39879
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab.
Entry for principal kudu/127.3.18.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab.
I20260928 20:19:43.743425 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.3:0
--local_ip_for_outbound_sockets=127.3.18.3
--webserver_interface=127.3.18.3
--webserver_port=0
--tserver_master_addrs=127.3.18.62:44021
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab
--principal=kudu/127.3.18.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:35033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:19:43.825441 7058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:19:43.825665 7058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:19:43.825690 7058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:19:43.826920 7058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:19:43.826968 7058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:19:43.826996 7058 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.3
I20260928 20:19:43.828603 7058 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:35033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab
--principal=kudu/127.3.18.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.18.3
--webserver_port=0
--tserver_master_addrs=127.3.18.62:44021
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.3
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:19:43.828890 7058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:19:43.829155 7058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 20:19:43.834995 7058 server_base.cc:1034] running on GCE node
W20260928 20:19:43.834982 7063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:43.835004 7064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:43.834981 7066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:43.835619 7058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:19:43.836047 7058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:19:43.837240 7058 hybrid_clock.cc:648] HybridClock initialized: now 1790626783837221 us; error 33 us; skew 500 ppm
Sep 28 20:19:43 dist-test-slave-b83v krb5kdc[6683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626783, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:19:43.840162 7058 init.cc:377] Logged in from keytab as kudu/127.3.18.3@KRBTEST.COM (short username kudu)
I20260928 20:19:43.840744 7058 webserver.cc:460] Webserver started at http://127.3.18.3:44217/ using document root <none> and password file <none>
I20260928 20:19:43.841004 7058 fs_manager.cc:359] Metadata directory not provided
I20260928 20:19:43.841074 7058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:19:43.841225 7058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:19:43.842335 7058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/minicluster-data/ts-2/data/instance:
uuid: "052d6f0ed4d44b679992e8059ade8c18"
format_stamp: "Formatted at 2026-09-28 20:19:43 on dist-test-slave-b83v"
server_key: "b14791ec75ad5813de87aabaf6cc3abd"
server_key_iv: "6aa789b1d86d879120cf80a193a0b737"
server_key_version: "encryptionkey@0"
I20260928 20:19:43.842715 7058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/minicluster-data/ts-2/wal/instance:
uuid: "052d6f0ed4d44b679992e8059ade8c18"
format_stamp: "Formatted at 2026-09-28 20:19:43 on dist-test-slave-b83v"
server_key: "b14791ec75ad5813de87aabaf6cc3abd"
server_key_iv: "6aa789b1d86d879120cf80a193a0b737"
server_key_version: "encryptionkey@0"
I20260928 20:19:43.844199 7058 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 20:19:43.845036 7073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:43.845229 7058 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:19:43.845392 7058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/minicluster-data/ts-2/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/minicluster-data/ts-2/wal
uuid: "052d6f0ed4d44b679992e8059ade8c18"
format_stamp: "Formatted at 2026-09-28 20:19:43 on dist-test-slave-b83v"
server_key: "b14791ec75ad5813de87aabaf6cc3abd"
server_key_iv: "6aa789b1d86d879120cf80a193a0b737"
server_key_version: "encryptionkey@0"
I20260928 20:19:43.845484 7058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:19:43.860347 7058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:19:43.862864 7058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:19:43.863049 7058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:19:43.863314 7058 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:19:43.863658 7058 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:19:43.863718 7058 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:43.863759 7058 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:19:43.863835 7058 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:43.869865 7058 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.3:37479
I20260928 20:19:43.869907 7186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.3:37479 every 8 connection(s)
I20260928 20:19:43.870227 7058 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
Sep 28 20:19:43 dist-test-slave-b83v krb5kdc[6683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626783, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.3@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:19:43.878640 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 7058
I20260928 20:19:43.878768 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790626764250411-3144-0/minicluster-data/ts-2/wal/instance
I20260928 20:19:43.878986 3144 external_mini_cluster.cc:1428] Setting key 9b6dbbc65f877239f4ad8090dce61097
I20260928 20:19:43.881184 6767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:43.870988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:58655 (local address 127.3.18.62:44021)
0928 20:19:43.871070 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:43.871072 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:43.871302 (+ 230us) server_negotiation.cc:407] Connection header received
0928 20:19:43.872434 (+ 1132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:43.872436 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:43.872447 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:43.872510 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:43.873314 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:43.873922 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:43.875334 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:43.875521 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:43.875580 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:43.877884 (+ 2304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:43.877886 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:43.877886 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:43.877893 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:43.879424 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:43.880066 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:43.880067 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:43.880068 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:43.880134 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:43.880413 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:43.880414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:43.880414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:43.880579 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:43.880645 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:43.881045 (+ 400us) server_negotiation.cc:299] Negotiation successful
0928 20:19:43.881117 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 20:19:43.881287 7189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:43.870976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:44021 (local address 127.3.18.3:58655)
0928 20:19:43.871166 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:43.871210 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:43.872326 (+ 1116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:43.872633 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:43.872637 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:43.872922 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:43.873224 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:43.873237 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:43.874034 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:43.874036 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:43.875252 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:43.875255 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:43.875602 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:43.875603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:43.875717 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:43.876325 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:19:43.876334 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:43.877764 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:43.879522 (+ 1758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:43.879530 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:43.879533 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:43.879974 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:43.880249 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:43.880250 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:43.880250 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:43.880361 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:43.880674 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:43.880680 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:43.880818 (+ 138us) client_negotiation.cc:769] Sending connection context
0928 20:19:43.880944 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 20:19:43.881087 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":108,"threads_started":1}
I20260928 20:19:43.881974 7187 heartbeater.cc:347] Connected to a master server at 127.3.18.62:44021
I20260928 20:19:43.882073 7187 heartbeater.cc:464] Registering TS with master...
I20260928 20:19:43.882283 7187 heartbeater.cc:511] Master 127.3.18.62:44021 requested a full tablet report, sending...
I20260928 20:19:43.882797 6728 ts_manager.cc:194] Registered new tserver with Master: 052d6f0ed4d44b679992e8059ade8c18 (127.3.18.3:37479)
I20260928 20:19:43.883177 6728 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.18.3@KRBTEST.COM'} at 127.3.18.3:58655
I20260928 20:19:43.891276 3144 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 20:19:43 dist-test-slave-b83v krb5kdc[6683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 20:19:43 dist-test-slave-b83v krb5kdc[6683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.18.1@KRBTEST.COM
I20260928 20:19:43.916744 7194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:43.907965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57674 (local address 127.3.18.1:46105)
0928 20:19:43.908128 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:43.908134 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:43.908155 (+ 21us) server_negotiation.cc:407] Connection header received
0928 20:19:43.908184 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:43.908186 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:43.908200 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:43.908273 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:43.908913 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:43.909709 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:43.910533 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:43.910883 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:43.910944 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:43.913598 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:43.913608 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:43.913608 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:43.913615 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:43.915343 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:43.915718 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:43.915722 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:43.915722 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:43.915826 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:43.916189 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:43.916190 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:43.916190 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:43.916385 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:43.916495 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:43.916588 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 20:19:43.916660 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":88,"threads_started":1}
Sep 28 20:19:43 dist-test-slave-b83v krb5kdc[6683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:19:43.925659 6767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:43.918211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47664 (local address 127.3.18.62:44021)
0928 20:19:43.918276 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:43.918278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:43.918293 (+ 15us) server_negotiation.cc:407] Connection header received
0928 20:19:43.918316 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:43.918317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:43.918325 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:43.918372 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:43.918842 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:43.919371 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:43.920224 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:43.920423 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:43.920489 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:43.922953 (+ 2464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:43.922955 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:43.922955 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:43.922969 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:43.924508 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:43.924861 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:43.924862 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:43.924862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:43.924920 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:43.925153 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:43.925153 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:43.925154 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:43.925307 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:43.925388 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:43.925510 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 20:19:43.925579 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260928 20:19:43.925941 6728 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47664
I20260928 20:19:43.926424 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 6790
I20260928 20:19:43.931116 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 6924
I20260928 20:19:43.935648 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 7058
I20260928 20:19:43.940188 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 6696
2026-09-28T20:19:43Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1886 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:19:44 dist-test-slave-b83v krb5kdc[7199](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:19:44 dist-test-slave-b83v krb5kdc[7199](info): set up 2 sockets
Sep 28 20:19:44 dist-test-slave-b83v krb5kdc[7199](info): commencing operation
krb5kdc: starting...
W20260928 20:19:44.996076 3144 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Sep 28 20:19:45 dist-test-slave-b83v krb5kdc[7199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:19:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:19:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.18.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab.
Entry for principal kudu/127.3.18.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab.
I20260928 20:19:45.119220 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:36113
--webserver_interface=127.3.18.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:40073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.18.62:36113
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:19:45.205919 7212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:19:45.206211 7212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:19:45.206267 7212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:19:45.207918 7212 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:19:45.208029 7212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:19:45.208060 7212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:19:45.208084 7212 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:19:45.208119 7212 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:19:45.209921 7212 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:40073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.18.62:36113
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:36113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.18.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:19:45.210289 7212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:19:45.210567 7212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:19:45.216063 7217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:45.216190 7212 server_base.cc:1034] running on GCE node
W20260928 20:19:45.216056 7218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:45.216056 7220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:45.216771 7212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:19:45.217180 7212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:19:45.218426 7212 hybrid_clock.cc:648] HybridClock initialized: now 1790626785218376 us; error 66 us; skew 500 ppm
Sep 28 20:19:45 dist-test-slave-b83v krb5kdc[7199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626785, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:19:45.221495 7212 init.cc:377] Logged in from keytab as kudu/127.3.18.62@KRBTEST.COM (short username kudu)
I20260928 20:19:45.221900 7212 fs_manager.cc:359] Metadata directory not provided
I20260928 20:19:45.221952 7212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:19:45.222096 7212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:19:45.223304 7212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/minicluster-data/master-0/data/instance:
uuid: "88e9b658fefb43dbbd2406d81beb5930"
format_stamp: "Formatted at 2026-09-28 20:19:45 on dist-test-slave-b83v"
server_key: "bde871744c32d454ff9bd31abc2fe09c"
server_key_iv: "eb1c7061962c99466a4396c4d4f9159d"
server_key_version: "encryptionkey@0"
I20260928 20:19:45.223711 7212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/minicluster-data/master-0/wal/instance:
uuid: "88e9b658fefb43dbbd2406d81beb5930"
format_stamp: "Formatted at 2026-09-28 20:19:45 on dist-test-slave-b83v"
server_key: "bde871744c32d454ff9bd31abc2fe09c"
server_key_iv: "eb1c7061962c99466a4396c4d4f9159d"
server_key_version: "encryptionkey@0"
I20260928 20:19:45.225263 7212 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:19:45.226183 7226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:45.226446 7212 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:19:45.226565 7212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/minicluster-data/master-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/minicluster-data/master-0/wal
uuid: "88e9b658fefb43dbbd2406d81beb5930"
format_stamp: "Formatted at 2026-09-28 20:19:45 on dist-test-slave-b83v"
server_key: "bde871744c32d454ff9bd31abc2fe09c"
server_key_iv: "eb1c7061962c99466a4396c4d4f9159d"
server_key_version: "encryptionkey@0"
I20260928 20:19:45.226649 7212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:19:45.256150 7212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:19:45.258721 7212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:19:45.258960 7212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:19:45.263319 7212 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.62:36113
I20260928 20:19:45.263487 7278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.62:36113 every 8 connection(s)
I20260928 20:19:45.263726 7212 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
I20260928 20:19:45.264878 7279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:45.265107 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 7212
I20260928 20:19:45.265228 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/minicluster-data/master-0/wal/instance
I20260928 20:19:45.265424 3144 external_mini_cluster.cc:1428] Setting key 97c25b5e6618fe7ed5b1f9309605cab6
I20260928 20:19:45.267663 7279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88e9b658fefb43dbbd2406d81beb5930: Bootstrap starting.
I20260928 20:19:45.268350 7279 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88e9b658fefb43dbbd2406d81beb5930: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:45.268651 7279 log.cc:824] T 00000000000000000000000000000000 P 88e9b658fefb43dbbd2406d81beb5930: Log is configured to *not* fsync() on all Append() calls
I20260928 20:19:45.269395 7279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88e9b658fefb43dbbd2406d81beb5930: No bootstrap required, opened a new log
I20260928 20:19:45.271242 7279 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88e9b658fefb43dbbd2406d81beb5930 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e9b658fefb43dbbd2406d81beb5930" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36113 } }
I20260928 20:19:45.271416 7279 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88e9b658fefb43dbbd2406d81beb5930 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:45.271468 7279 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88e9b658fefb43dbbd2406d81beb5930 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88e9b658fefb43dbbd2406d81beb5930, State: Initialized, Role: FOLLOWER
I20260928 20:19:45.271637 7279 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88e9b658fefb43dbbd2406d81beb5930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e9b658fefb43dbbd2406d81beb5930" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36113 } }
I20260928 20:19:45.271739 7279 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88e9b658fefb43dbbd2406d81beb5930 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:19:45.271803 7279 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88e9b658fefb43dbbd2406d81beb5930 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:19:45.271847 7279 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88e9b658fefb43dbbd2406d81beb5930 [term 0 FOLLOWER]: Advancing to term 1
Sep 28 20:19:45 dist-test-slave-b83v krb5kdc[7199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:19:45.272720 7279 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88e9b658fefb43dbbd2406d81beb5930 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e9b658fefb43dbbd2406d81beb5930" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36113 } }
I20260928 20:19:45.272850 7279 leader_election.cc:302] T 00000000000000000000000000000000 P 88e9b658fefb43dbbd2406d81beb5930 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88e9b658fefb43dbbd2406d81beb5930; no voters:
I20260928 20:19:45.273046 7279 leader_election.cc:288] T 00000000000000000000000000000000 P 88e9b658fefb43dbbd2406d81beb5930 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:19:45.273123 7284 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88e9b658fefb43dbbd2406d81beb5930 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:19:45.273284 7284 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88e9b658fefb43dbbd2406d81beb5930 [term 1 LEADER]: Becoming Leader. State: Replica: 88e9b658fefb43dbbd2406d81beb5930, State: Running, Role: LEADER
I20260928 20:19:45.273382 7279 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88e9b658fefb43dbbd2406d81beb5930 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:19:45.273408 7284 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88e9b658fefb43dbbd2406d81beb5930 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e9b658fefb43dbbd2406d81beb5930" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36113 } }
I20260928 20:19:45.273769 7286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88e9b658fefb43dbbd2406d81beb5930 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88e9b658fefb43dbbd2406d81beb5930" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e9b658fefb43dbbd2406d81beb5930" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36113 } } }
I20260928 20:19:45.273814 7284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88e9b658fefb43dbbd2406d81beb5930 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e9b658fefb43dbbd2406d81beb5930. Latest consensus state: current_term: 1 leader_uuid: "88e9b658fefb43dbbd2406d81beb5930" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e9b658fefb43dbbd2406d81beb5930" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36113 } } }
I20260928 20:19:45.273876 7286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88e9b658fefb43dbbd2406d81beb5930 [sys.catalog]: This master's current role is: LEADER
I20260928 20:19:45.273881 7284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88e9b658fefb43dbbd2406d81beb5930 [sys.catalog]: This master's current role is: LEADER
I20260928 20:19:45.274636 7290 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:19:45.275161 7290 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:19:45.276942 7290 catalog_manager.cc:1347] Generated new cluster ID: a34e25a8155d4a42809f01241a121758
I20260928 20:19:45.277000 7290 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:19:45.277616 7282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:45.266044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42664 (local address 127.3.18.62:36113)
0928 20:19:45.266230 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:45.266242 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:45.266328 (+ 86us) server_negotiation.cc:407] Connection header received
0928 20:19:45.266824 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:45.266831 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:45.266884 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:45.267067 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:45.267716 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:45.268773 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:45.269608 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:45.270018 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:45.270150 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:45.272976 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:45.272981 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:45.272982 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:45.272997 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:45.275612 (+ 2615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:45.276118 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:45.276122 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:45.276122 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:45.276217 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:45.276486 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:45.276487 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:45.276488 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:45.276830 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:45.276969 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:45.277234 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 20:19:45.277412 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":96,"threads_started":1}
I20260928 20:19:45.287019 7290 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:19:45.287621 7290 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:19:45.296958 7290 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88e9b658fefb43dbbd2406d81beb5930: Generated new TSK 0
I20260928 20:19:45.297235 7290 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.18.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab.
Entry for principal kudu/127.3.18.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab.
I20260928 20:19:45.328285 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.1:0
--local_ip_for_outbound_sockets=127.3.18.1
--webserver_interface=127.3.18.1
--webserver_port=0
--tserver_master_addrs=127.3.18.62:36113
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab
--principal=kudu/127.3.18.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:40073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:19:45.414726 7305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:19:45.414950 7305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:19:45.414978 7305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:19:45.416186 7305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:19:45.416229 7305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:19:45.416260 7305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.1
I20260928 20:19:45.417781 7305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:40073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab
--principal=kudu/127.3.18.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.18.1
--webserver_port=0
--tserver_master_addrs=127.3.18.62:36113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.1
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:19:45.417994 7305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:19:45.418179 7305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:19:45.424005 7313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:45.424001 7310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:45.424021 7305 server_base.cc:1034] running on GCE node
W20260928 20:19:45.424001 7311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:45.424609 7305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:19:45.424903 7305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:19:45.426056 7305 hybrid_clock.cc:648] HybridClock initialized: now 1790626785426038 us; error 43 us; skew 500 ppm
Sep 28 20:19:45 dist-test-slave-b83v krb5kdc[7199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626785, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:19:45.428890 7305 init.cc:377] Logged in from keytab as kudu/127.3.18.1@KRBTEST.COM (short username kudu)
I20260928 20:19:45.429337 7305 fs_manager.cc:359] Metadata directory not provided
I20260928 20:19:45.429389 7305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:19:45.429533 7305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:19:45.430749 7305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/minicluster-data/ts-0/data/instance:
uuid: "d224f54e51a24227a6b7a34e87d8e0b0"
format_stamp: "Formatted at 2026-09-28 20:19:45 on dist-test-slave-b83v"
server_key: "f1588d9a2e34ca8b677a02f3c1829442"
server_key_iv: "bd00a4daa3f80c445f9fa1bfc520b59a"
server_key_version: "encryptionkey@0"
I20260928 20:19:45.431185 7305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/minicluster-data/ts-0/wal/instance:
uuid: "d224f54e51a24227a6b7a34e87d8e0b0"
format_stamp: "Formatted at 2026-09-28 20:19:45 on dist-test-slave-b83v"
server_key: "f1588d9a2e34ca8b677a02f3c1829442"
server_key_iv: "bd00a4daa3f80c445f9fa1bfc520b59a"
server_key_version: "encryptionkey@0"
I20260928 20:19:45.432632 7305 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:19:45.433610 7319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:45.433859 7305 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:19:45.433960 7305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/minicluster-data/ts-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/minicluster-data/ts-0/wal
uuid: "d224f54e51a24227a6b7a34e87d8e0b0"
format_stamp: "Formatted at 2026-09-28 20:19:45 on dist-test-slave-b83v"
server_key: "f1588d9a2e34ca8b677a02f3c1829442"
server_key_iv: "bd00a4daa3f80c445f9fa1bfc520b59a"
server_key_version: "encryptionkey@0"
I20260928 20:19:45.434037 7305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:19:45.446345 7305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:19:45.448575 7305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:19:45.448691 7305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:19:45.449014 7305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:19:45.449354 7305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:19:45.449391 7305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:45.449414 7305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:19:45.449426 7305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:45.455868 7305 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.1:34125
I20260928 20:19:45.455883 7432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.1:34125 every 8 connection(s)
I20260928 20:19:45.456251 7305 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
Sep 28 20:19:45 dist-test-slave-b83v krb5kdc[7199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626785, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.1@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:19:45.463961 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 7305
I20260928 20:19:45.464099 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/minicluster-data/ts-0/wal/instance
I20260928 20:19:45.464334 3144 external_mini_cluster.cc:1428] Setting key db72a7b0041ee0a14d5028d9eba8be68
I20260928 20:19:45.467893 7282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:45.457194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:39397 (local address 127.3.18.62:36113)
0928 20:19:45.457283 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:45.457286 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:45.457598 (+ 312us) server_negotiation.cc:407] Connection header received
0928 20:19:45.458116 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:45.458118 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:45.458130 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:45.458192 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:45.459289 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:45.460002 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:45.461143 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:45.461390 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:45.461465 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:45.464049 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:45.464052 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:45.464052 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:45.464062 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:45.465834 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:45.466498 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:45.466500 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:45.466501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:45.466584 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:45.466939 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:45.466939 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:45.466940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:45.467111 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:45.467199 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:45.467612 (+ 413us) server_negotiation.cc:299] Negotiation successful
0928 20:19:45.467733 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 20:19:45.468146 7435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:45.457124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36113 (local address 127.3.18.1:39397)
0928 20:19:45.457365 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:45.457419 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:45.457997 (+ 578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:45.458267 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:45.458271 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:45.458620 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:45.459159 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:45.459176 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:45.460096 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:45.460097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:45.461019 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:45.461022 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:45.461501 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:45.461502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:45.461615 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:45.462235 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:19:45.462245 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:45.463928 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:45.465951 (+ 2023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:45.465964 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:45.465967 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:45.466349 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:45.466690 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:45.466691 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:45.466692 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:45.466835 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:45.467226 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:45.467255 (+ 29us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:45.467437 (+ 182us) client_negotiation.cc:769] Sending connection context
0928 20:19:45.467561 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 20:19:45.467734 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":91,"threads_started":1}
I20260928 20:19:45.469046 7433 heartbeater.cc:347] Connected to a master server at 127.3.18.62:36113
I20260928 20:19:45.469201 7433 heartbeater.cc:464] Registering TS with master...
I20260928 20:19:45.469417 7433 heartbeater.cc:511] Master 127.3.18.62:36113 requested a full tablet report, sending...
I20260928 20:19:45.470145 7243 ts_manager.cc:194] Registered new tserver with Master: d224f54e51a24227a6b7a34e87d8e0b0 (127.3.18.1:34125)
I20260928 20:19:45.470888 7243 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.18.1@KRBTEST.COM'} at 127.3.18.1:39397
WARNING: no policy specified for kudu/127.3.18.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab.
Entry for principal kudu/127.3.18.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab.
I20260928 20:19:45.496073 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.2:0
--local_ip_for_outbound_sockets=127.3.18.2
--webserver_interface=127.3.18.2
--webserver_port=0
--tserver_master_addrs=127.3.18.62:36113
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab
--principal=kudu/127.3.18.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:40073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:19:45.585134 7438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:19:45.585434 7438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:19:45.585481 7438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:19:45.586822 7438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:19:45.586910 7438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:19:45.586982 7438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.2
I20260928 20:19:45.588721 7438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:40073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab
--principal=kudu/127.3.18.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.18.2
--webserver_port=0
--tserver_master_addrs=127.3.18.62:36113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.2
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:19:45.589036 7438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:19:45.589298 7438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:19:45.594714 7446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:45.594714 7444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:45.594828 7443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:45.594952 7438 server_base.cc:1034] running on GCE node
I20260928 20:19:45.595163 7438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:19:45.595525 7438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:19:45.596719 7438 hybrid_clock.cc:648] HybridClock initialized: now 1790626785596721 us; error 45 us; skew 500 ppm
Sep 28 20:19:45 dist-test-slave-b83v krb5kdc[7199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626785, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:19:45.599457 7438 init.cc:377] Logged in from keytab as kudu/127.3.18.2@KRBTEST.COM (short username kudu)
I20260928 20:19:45.599913 7438 fs_manager.cc:359] Metadata directory not provided
I20260928 20:19:45.599969 7438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:19:45.600067 7438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:19:45.601152 7438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/minicluster-data/ts-1/data/instance:
uuid: "d46718e3c33c4be79d574901e48a5eeb"
format_stamp: "Formatted at 2026-09-28 20:19:45 on dist-test-slave-b83v"
server_key: "85121493c7578aab6b5f04bd97a4a97a"
server_key_iv: "8b5abb266914f0f8875357319ff8752a"
server_key_version: "encryptionkey@0"
I20260928 20:19:45.601526 7438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/minicluster-data/ts-1/wal/instance:
uuid: "d46718e3c33c4be79d574901e48a5eeb"
format_stamp: "Formatted at 2026-09-28 20:19:45 on dist-test-slave-b83v"
server_key: "85121493c7578aab6b5f04bd97a4a97a"
server_key_iv: "8b5abb266914f0f8875357319ff8752a"
server_key_version: "encryptionkey@0"
I20260928 20:19:45.602851 7438 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:19:45.603677 7452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:45.603901 7438 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:19:45.603991 7438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/minicluster-data/ts-1/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/minicluster-data/ts-1/wal
uuid: "d46718e3c33c4be79d574901e48a5eeb"
format_stamp: "Formatted at 2026-09-28 20:19:45 on dist-test-slave-b83v"
server_key: "85121493c7578aab6b5f04bd97a4a97a"
server_key_iv: "8b5abb266914f0f8875357319ff8752a"
server_key_version: "encryptionkey@0"
I20260928 20:19:45.604049 7438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:19:45.632926 7438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:19:45.635325 7438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:19:45.635452 7438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:19:45.635633 7438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:19:45.636001 7438 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:19:45.636039 7438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:45.636062 7438 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:19:45.636111 7438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:45.642148 7438 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.2:37435
I20260928 20:19:45.642221 7565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.2:37435 every 8 connection(s)
I20260928 20:19:45.642524 7438 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
Sep 28 20:19:45 dist-test-slave-b83v krb5kdc[7199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626785, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.2@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:19:45.652134 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 7438
I20260928 20:19:45.652253 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/minicluster-data/ts-1/wal/instance
I20260928 20:19:45.652454 7282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:45.643356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:53847 (local address 127.3.18.62:36113)
0928 20:19:45.643444 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:45.643446 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:45.643622 (+ 176us) server_negotiation.cc:407] Connection header received
0928 20:19:45.644187 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:45.644188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:45.644198 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:45.644264 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:45.645225 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:45.645784 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:45.646822 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:45.647027 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:45.647083 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:45.649287 (+ 2204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:45.649288 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:45.649289 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:45.649296 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:45.650803 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:45.651189 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:45.651190 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:45.651191 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:45.651238 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:45.651522 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:45.651524 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:45.651524 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:45.651645 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:45.651710 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:45.652174 (+ 464us) server_negotiation.cc:299] Negotiation successful
0928 20:19:45.652386 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 20:19:45.652515 3144 external_mini_cluster.cc:1428] Setting key af383eb9ed7da08141752e97bd8e8350
I20260928 20:19:45.652581 7568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:45.643272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36113 (local address 127.3.18.2:53847)
0928 20:19:45.643469 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:45.643510 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:45.644059 (+ 549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:45.644350 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:45.644354 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:45.644609 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:45.645129 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:45.645134 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:45.645901 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:45.645902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:45.646738 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:45.646741 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:45.647103 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:45.647103 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:45.647202 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:45.647795 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:19:45.647806 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:45.649203 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:45.650891 (+ 1688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:45.650897 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:45.650899 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:45.651122 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:45.651298 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:45.651298 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:45.651299 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:45.651426 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:45.651721 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:45.651727 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:45.651994 (+ 267us) client_negotiation.cc:769] Sending connection context
0928 20:19:45.652133 (+ 139us) client_negotiation.cc:240] Negotiation successful
0928 20:19:45.652322 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":100,"threads_started":1}
I20260928 20:19:45.653128 7566 heartbeater.cc:347] Connected to a master server at 127.3.18.62:36113
I20260928 20:19:45.653249 7566 heartbeater.cc:464] Registering TS with master...
I20260928 20:19:45.653414 7566 heartbeater.cc:511] Master 127.3.18.62:36113 requested a full tablet report, sending...
I20260928 20:19:45.653813 7243 ts_manager.cc:194] Registered new tserver with Master: d46718e3c33c4be79d574901e48a5eeb (127.3.18.2:37435)
I20260928 20:19:45.654198 7243 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.18.2@KRBTEST.COM'} at 127.3.18.2:53847
WARNING: no policy specified for kudu/127.3.18.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab.
Entry for principal kudu/127.3.18.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab.
I20260928 20:19:45.680332 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.3:0
--local_ip_for_outbound_sockets=127.3.18.3
--webserver_interface=127.3.18.3
--webserver_port=0
--tserver_master_addrs=127.3.18.62:36113
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab
--principal=kudu/127.3.18.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:40073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:19:45.766077 7571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:19:45.766278 7571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:19:45.766300 7571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:19:45.767503 7571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:19:45.767553 7571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:19:45.767580 7571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.3
I20260928 20:19:45.769136 7571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:40073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab
--principal=kudu/127.3.18.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.18.3
--webserver_port=0
--tserver_master_addrs=127.3.18.62:36113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.3
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:19:45.769397 7571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:19:45.769662 7571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:19:45.774744 7579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:45.774986 7577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:45.775014 7576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:45.775197 7571 server_base.cc:1034] running on GCE node
I20260928 20:19:45.775584 7571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:19:45.775924 7571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:19:45.777119 7571 hybrid_clock.cc:648] HybridClock initialized: now 1790626785777117 us; error 43 us; skew 500 ppm
Sep 28 20:19:45 dist-test-slave-b83v krb5kdc[7199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626785, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:19:45.779958 7571 init.cc:377] Logged in from keytab as kudu/127.3.18.3@KRBTEST.COM (short username kudu)
I20260928 20:19:45.780355 7571 fs_manager.cc:359] Metadata directory not provided
I20260928 20:19:45.780450 7571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:19:45.780577 7571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:19:45.781630 7571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/minicluster-data/ts-2/data/instance:
uuid: "8c8538aa487949c6905f36a294448ce4"
format_stamp: "Formatted at 2026-09-28 20:19:45 on dist-test-slave-b83v"
server_key: "224276c35e8ba4f60e2b59b092be9dff"
server_key_iv: "280cbdad1f7f554bd691d42553f12c15"
server_key_version: "encryptionkey@0"
I20260928 20:19:45.781929 7571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/minicluster-data/ts-2/wal/instance:
uuid: "8c8538aa487949c6905f36a294448ce4"
format_stamp: "Formatted at 2026-09-28 20:19:45 on dist-test-slave-b83v"
server_key: "224276c35e8ba4f60e2b59b092be9dff"
server_key_iv: "280cbdad1f7f554bd691d42553f12c15"
server_key_version: "encryptionkey@0"
I20260928 20:19:45.783164 7571 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 20:19:45.783980 7585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:45.784240 7571 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:19:45.784355 7571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/minicluster-data/ts-2/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/minicluster-data/ts-2/wal
uuid: "8c8538aa487949c6905f36a294448ce4"
format_stamp: "Formatted at 2026-09-28 20:19:45 on dist-test-slave-b83v"
server_key: "224276c35e8ba4f60e2b59b092be9dff"
server_key_iv: "280cbdad1f7f554bd691d42553f12c15"
server_key_version: "encryptionkey@0"
I20260928 20:19:45.784444 7571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:19:45.791847 7571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:19:45.794144 7571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:19:45.794366 7571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:19:45.794612 7571 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:19:45.794958 7571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:19:45.794996 7571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:45.795018 7571 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:19:45.795065 7571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:45.801090 7571 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.3:33577
I20260928 20:19:45.801164 7698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.3:33577 every 8 connection(s)
I20260928 20:19:45.801446 7571 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
I20260928 20:19:45.805903 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 7571
I20260928 20:19:45.806013 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/minicluster-data/ts-2/wal/instance
I20260928 20:19:45.806216 3144 external_mini_cluster.cc:1428] Setting key 08685ce974a18edc2401739ab894b7d5
Sep 28 20:19:45 dist-test-slave-b83v krb5kdc[7199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626785, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.3@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:19:45.811474 7282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:45.802171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:58753 (local address 127.3.18.62:36113)
0928 20:19:45.802229 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:45.802231 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:45.802406 (+ 175us) server_negotiation.cc:407] Connection header received
0928 20:19:45.802895 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:45.802896 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:45.802907 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:45.802975 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:45.803961 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:45.804466 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:45.805478 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:45.805664 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:45.805724 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:45.808513 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:45.808514 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:45.808515 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:45.808522 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:45.809920 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:45.810380 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:45.810381 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:45.810382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:45.810487 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:45.810744 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:45.810745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:45.810745 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:45.810867 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:45.810927 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:45.811292 (+ 365us) server_negotiation.cc:299] Negotiation successful
0928 20:19:45.811365 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 20:19:45.811746 7701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:45.802120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36113 (local address 127.3.18.3:58753)
0928 20:19:45.802278 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:45.802320 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:45.802790 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:45.803032 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:45.803036 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:45.803310 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:45.803868 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:45.803879 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:45.804566 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:45.804567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:45.805392 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:45.805395 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:45.805852 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:45.805853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:45.806021 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:45.806659 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:19:45.806669 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:45.808414 (+ 1745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:45.810013 (+ 1599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:45.810020 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:45.810022 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:45.810250 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:45.810563 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:45.810564 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:45.810564 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:45.810671 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:45.810967 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:45.810973 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:45.811128 (+ 155us) client_negotiation.cc:769] Sending connection context
0928 20:19:45.811266 (+ 138us) client_negotiation.cc:240] Negotiation successful
0928 20:19:45.811443 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":91,"threads_started":1}
I20260928 20:19:45.812505 7699 heartbeater.cc:347] Connected to a master server at 127.3.18.62:36113
I20260928 20:19:45.812609 7699 heartbeater.cc:464] Registering TS with master...
I20260928 20:19:45.812793 7699 heartbeater.cc:511] Master 127.3.18.62:36113 requested a full tablet report, sending...
I20260928 20:19:45.813202 7243 ts_manager.cc:194] Registered new tserver with Master: 8c8538aa487949c6905f36a294448ce4 (127.3.18.3:33577)
I20260928 20:19:45.813633 7243 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.18.3@KRBTEST.COM'} at 127.3.18.3:58753
I20260928 20:19:45.818029 3144 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 20:19:45.826212 7282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:45.819522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42674 (local address 127.3.18.62:36113)
0928 20:19:45.819587 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:45.819589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:45.819708 (+ 119us) server_negotiation.cc:407] Connection header received
0928 20:19:45.819810 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:45.819811 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:45.819823 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:45.819881 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:45.820568 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:45.821114 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:45.821986 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:45.822179 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:45.822238 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:45.823529 (+ 1291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:45.823531 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:45.823531 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:45.823538 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:45.824993 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:45.825415 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:45.825416 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:45.825416 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:45.825472 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:45.825704 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:45.825704 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:45.825704 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:45.825862 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:45.825939 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:45.826073 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 20:19:45.826135 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 20:19:45.828439 7243 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42674:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 20:19:45.830143 7243 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 20:19:45.837352 7714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:45.832860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.1:34125 (local address 127.0.0.1:36948)
0928 20:19:45.833251 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:45.833261 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:45.833303 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:45.833489 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:45.833493 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:45.833495 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:19:45.833577 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:45.833580 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:45.834707 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:45.834708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:45.836045 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:45.836048 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:45.836980 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:45.836981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:45.837073 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:45.837088 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 20:19:45.837191 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 20:19:45.837257 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"mutex_wait_us":2,"thread_start_us":81,"threads_started":1}
I20260928 20:19:45.837363 7715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:45.833132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36948 (local address 127.3.18.1:34125)
0928 20:19:45.833308 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:45.833313 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:45.833324 (+ 11us) server_negotiation.cc:407] Connection header received
0928 20:19:45.833359 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:45.833361 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:45.833382 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:45.833454 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:19:45.833656 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:45.834602 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:45.836143 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:45.836863 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:45.836976 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:45.837049 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:45.837212 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 20:19:45.837257 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":69,"threads_started":1}
I20260928 20:19:45.837818 7711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:45.832860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.3:33577 (local address 127.0.0.1:53476)
0928 20:19:45.833678 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:45.833688 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:45.833742 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:45.834022 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:45.834023 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:45.834025 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:19:45.834074 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:45.834075 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:45.835345 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:45.835346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:45.836501 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:45.836504 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:45.837564 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:45.837565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:45.837622 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:45.837625 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 20:19:45.837677 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 20:19:45.837727 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":800,"mutex_wait_us":254,"thread_start_us":79,"threads_started":1}
I20260928 20:19:45.837850 7712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:45.832958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53476 (local address 127.3.18.3:33577)
0928 20:19:45.833350 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:45.833355 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:45.833767 (+ 412us) server_negotiation.cc:407] Connection header received
0928 20:19:45.833821 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:45.833823 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:45.833840 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:45.833930 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:19:45.834166 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:45.835238 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:45.836595 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:45.837365 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:45.837432 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:45.837533 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:45.837703 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 20:19:45.837757 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":98,"threads_started":1}
I20260928 20:19:45.837831 7710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:45.832860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.2:37435 (local address 127.0.0.1:49380)
0928 20:19:45.833059 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:45.833109 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:45.833187 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:45.833849 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:45.833850 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:45.833852 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:19:45.833895 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:45.833897 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:45.835189 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:45.835190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:45.836247 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:45.836249 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:45.837280 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:45.837281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:45.837360 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:45.837373 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 20:19:45.837728 (+ 355us) client_negotiation.cc:240] Negotiation successful
0928 20:19:45.837764 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":56,"threads_started":1}
I20260928 20:19:45.838482 7713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:45.833107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49380 (local address 127.3.18.2:37435)
0928 20:19:45.833509 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:45.833513 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:45.833523 (+ 10us) server_negotiation.cc:407] Connection header received
0928 20:19:45.833559 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:45.833562 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:45.833578 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:45.833853 (+ 275us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:19:45.833950 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:45.835074 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:45.836394 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:45.837168 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:45.837920 (+ 752us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:45.838308 (+ 388us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:45.838336 (+ 28us) server_negotiation.cc:299] Negotiation successful
0928 20:19:45.838395 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":80,"threads_started":1}
I20260928 20:19:45.838455 7367 tablet_service.cc:1467] Processing CreateTablet for tablet 78821955484a4838be8af47487dbe815 (DEFAULT_TABLE table=test-table [id=abd2cfe6d6164d1bbe870f7e47a9fd5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:19:45.838685 7633 tablet_service.cc:1467] Processing CreateTablet for tablet 78821955484a4838be8af47487dbe815 (DEFAULT_TABLE table=test-table [id=abd2cfe6d6164d1bbe870f7e47a9fd5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:19:45.838852 7367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78821955484a4838be8af47487dbe815. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:45.838992 7633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78821955484a4838be8af47487dbe815. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:45.839152 7500 tablet_service.cc:1467] Processing CreateTablet for tablet 78821955484a4838be8af47487dbe815 (DEFAULT_TABLE table=test-table [id=abd2cfe6d6164d1bbe870f7e47a9fd5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:19:45.839452 7500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78821955484a4838be8af47487dbe815. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:45.841928 7716 tablet_bootstrap.cc:492] T 78821955484a4838be8af47487dbe815 P 8c8538aa487949c6905f36a294448ce4: Bootstrap starting.
I20260928 20:19:45.842463 7717 tablet_bootstrap.cc:492] T 78821955484a4838be8af47487dbe815 P d224f54e51a24227a6b7a34e87d8e0b0: Bootstrap starting.
I20260928 20:19:45.842752 7716 tablet_bootstrap.cc:654] T 78821955484a4838be8af47487dbe815 P 8c8538aa487949c6905f36a294448ce4: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:45.843129 7718 tablet_bootstrap.cc:492] T 78821955484a4838be8af47487dbe815 P d46718e3c33c4be79d574901e48a5eeb: Bootstrap starting.
I20260928 20:19:45.843165 7717 tablet_bootstrap.cc:654] T 78821955484a4838be8af47487dbe815 P d224f54e51a24227a6b7a34e87d8e0b0: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:45.843505 7716 log.cc:824] T 78821955484a4838be8af47487dbe815 P 8c8538aa487949c6905f36a294448ce4: Log is configured to *not* fsync() on all Append() calls
I20260928 20:19:45.843883 7718 tablet_bootstrap.cc:654] T 78821955484a4838be8af47487dbe815 P d46718e3c33c4be79d574901e48a5eeb: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:45.844041 7717 log.cc:824] T 78821955484a4838be8af47487dbe815 P d224f54e51a24227a6b7a34e87d8e0b0: Log is configured to *not* fsync() on all Append() calls
I20260928 20:19:45.844539 7718 log.cc:824] T 78821955484a4838be8af47487dbe815 P d46718e3c33c4be79d574901e48a5eeb: Log is configured to *not* fsync() on all Append() calls
I20260928 20:19:45.844707 7716 tablet_bootstrap.cc:492] T 78821955484a4838be8af47487dbe815 P 8c8538aa487949c6905f36a294448ce4: No bootstrap required, opened a new log
I20260928 20:19:45.844792 7716 ts_tablet_manager.cc:1397] T 78821955484a4838be8af47487dbe815 P 8c8538aa487949c6905f36a294448ce4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 20:19:45.844880 7717 tablet_bootstrap.cc:492] T 78821955484a4838be8af47487dbe815 P d224f54e51a24227a6b7a34e87d8e0b0: No bootstrap required, opened a new log
I20260928 20:19:45.844954 7717 ts_tablet_manager.cc:1397] T 78821955484a4838be8af47487dbe815 P d224f54e51a24227a6b7a34e87d8e0b0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 20:19:45.845304 7718 tablet_bootstrap.cc:492] T 78821955484a4838be8af47487dbe815 P d46718e3c33c4be79d574901e48a5eeb: No bootstrap required, opened a new log
I20260928 20:19:45.845360 7718 ts_tablet_manager.cc:1397] T 78821955484a4838be8af47487dbe815 P d46718e3c33c4be79d574901e48a5eeb: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 20:19:45.846273 7716 raft_consensus.cc:348] T 78821955484a4838be8af47487dbe815 P 8c8538aa487949c6905f36a294448ce4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d224f54e51a24227a6b7a34e87d8e0b0" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34125 } } peers { permanent_uuid: "8c8538aa487949c6905f36a294448ce4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 33577 } } peers { permanent_uuid: "d46718e3c33c4be79d574901e48a5eeb" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 37435 } }
I20260928 20:19:45.846431 7716 raft_consensus.cc:374] T 78821955484a4838be8af47487dbe815 P 8c8538aa487949c6905f36a294448ce4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:45.846454 7716 raft_consensus.cc:729] T 78821955484a4838be8af47487dbe815 P 8c8538aa487949c6905f36a294448ce4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c8538aa487949c6905f36a294448ce4, State: Initialized, Role: FOLLOWER
I20260928 20:19:45.846596 7716 consensus_queue.cc:260] T 78821955484a4838be8af47487dbe815 P 8c8538aa487949c6905f36a294448ce4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d224f54e51a24227a6b7a34e87d8e0b0" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34125 } } peers { permanent_uuid: "8c8538aa487949c6905f36a294448ce4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 33577 } } peers { permanent_uuid: "d46718e3c33c4be79d574901e48a5eeb" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 37435 } }
I20260928 20:19:45.847216 7716 ts_tablet_manager.cc:1428] T 78821955484a4838be8af47487dbe815 P 8c8538aa487949c6905f36a294448ce4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 20:19:45.847190 7717 raft_consensus.cc:348] T 78821955484a4838be8af47487dbe815 P d224f54e51a24227a6b7a34e87d8e0b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d224f54e51a24227a6b7a34e87d8e0b0" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34125 } } peers { permanent_uuid: "8c8538aa487949c6905f36a294448ce4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 33577 } } peers { permanent_uuid: "d46718e3c33c4be79d574901e48a5eeb" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 37435 } }
I20260928 20:19:45.847376 7699 heartbeater.cc:503] Master 127.3.18.62:36113 was elected leader, sending a full tablet report...
I20260928 20:19:45.847375 7717 raft_consensus.cc:374] T 78821955484a4838be8af47487dbe815 P d224f54e51a24227a6b7a34e87d8e0b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:45.847429 7717 raft_consensus.cc:729] T 78821955484a4838be8af47487dbe815 P d224f54e51a24227a6b7a34e87d8e0b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d224f54e51a24227a6b7a34e87d8e0b0, State: Initialized, Role: FOLLOWER
I20260928 20:19:45.847440 7718 raft_consensus.cc:348] T 78821955484a4838be8af47487dbe815 P d46718e3c33c4be79d574901e48a5eeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d224f54e51a24227a6b7a34e87d8e0b0" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34125 } } peers { permanent_uuid: "8c8538aa487949c6905f36a294448ce4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 33577 } } peers { permanent_uuid: "d46718e3c33c4be79d574901e48a5eeb" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 37435 } }
I20260928 20:19:45.847581 7718 raft_consensus.cc:374] T 78821955484a4838be8af47487dbe815 P d46718e3c33c4be79d574901e48a5eeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:45.847612 7718 raft_consensus.cc:729] T 78821955484a4838be8af47487dbe815 P d46718e3c33c4be79d574901e48a5eeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d46718e3c33c4be79d574901e48a5eeb, State: Initialized, Role: FOLLOWER
I20260928 20:19:45.847555 7717 consensus_queue.cc:260] T 78821955484a4838be8af47487dbe815 P d224f54e51a24227a6b7a34e87d8e0b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d224f54e51a24227a6b7a34e87d8e0b0" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34125 } } peers { permanent_uuid: "8c8538aa487949c6905f36a294448ce4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 33577 } } peers { permanent_uuid: "d46718e3c33c4be79d574901e48a5eeb" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 37435 } }
I20260928 20:19:45.847716 7718 consensus_queue.cc:260] T 78821955484a4838be8af47487dbe815 P d46718e3c33c4be79d574901e48a5eeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d224f54e51a24227a6b7a34e87d8e0b0" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34125 } } peers { permanent_uuid: "8c8538aa487949c6905f36a294448ce4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 33577 } } peers { permanent_uuid: "d46718e3c33c4be79d574901e48a5eeb" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 37435 } }
I20260928 20:19:45.847900 7717 ts_tablet_manager.cc:1428] T 78821955484a4838be8af47487dbe815 P d224f54e51a24227a6b7a34e87d8e0b0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 20:19:45.848018 7718 ts_tablet_manager.cc:1428] T 78821955484a4838be8af47487dbe815 P d46718e3c33c4be79d574901e48a5eeb: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260928 20:19:45.848027 7433 heartbeater.cc:503] Master 127.3.18.62:36113 was elected leader, sending a full tablet report...
I20260928 20:19:45.848117 7566 heartbeater.cc:503] Master 127.3.18.62:36113 was elected leader, sending a full tablet report...
I20260928 20:19:45.888185 7723 raft_consensus.cc:482] T 78821955484a4838be8af47487dbe815 P d224f54e51a24227a6b7a34e87d8e0b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:19:45.888384 7723 raft_consensus.cc:504] T 78821955484a4838be8af47487dbe815 P d224f54e51a24227a6b7a34e87d8e0b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d224f54e51a24227a6b7a34e87d8e0b0" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34125 } } peers { permanent_uuid: "8c8538aa487949c6905f36a294448ce4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 33577 } } peers { permanent_uuid: "d46718e3c33c4be79d574901e48a5eeb" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 37435 } }
I20260928 20:19:45.888754 7723 leader_election.cc:288] T 78821955484a4838be8af47487dbe815 P d224f54e51a24227a6b7a34e87d8e0b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c8538aa487949c6905f36a294448ce4 (127.3.18.3:33577), d46718e3c33c4be79d574901e48a5eeb (127.3.18.2:37435)
I20260928 20:19:45.892946 7713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:45.889079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:37677 (local address 127.3.18.2:37435)
0928 20:19:45.889161 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:45.889164 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:45.889177 (+ 13us) server_negotiation.cc:407] Connection header received
0928 20:19:45.889210 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:45.889211 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:45.889223 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:45.889287 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:19:45.889505 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:45.890246 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:45.891923 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:45.892499 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:45.892581 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:45.892593 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:45.892824 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 20:19:45.892870 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 20:19:45.892963 7712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:45.889093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:35891 (local address 127.3.18.3:33577)
0928 20:19:45.889147 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:45.889149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:45.889360 (+ 211us) server_negotiation.cc:407] Connection header received
0928 20:19:45.889394 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:45.889395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:45.889406 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:45.889480 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:19:45.889577 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:45.890283 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:45.891906 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:45.892499 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:45.892581 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:45.892593 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:45.892824 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 20:19:45.892869 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 20:19:45.892974 7435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:45.889000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.2:37435 (local address 127.3.18.1:37677)
0928 20:19:45.889074 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:45.889086 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:45.889134 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:45.889319 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:45.889320 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:45.889323 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:19:45.889399 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:45.889401 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:45.890381 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:45.890383 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:45.891843 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:45.891845 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:45.892625 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:45.892627 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:45.892726 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:45.892733 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 20:19:45.892803 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 20:19:45.892854 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":103}
W20260928 20:19:45.893183 7567 tablet.cc:2369] T 78821955484a4838be8af47487dbe815 P d46718e3c33c4be79d574901e48a5eeb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 20:19:45.892980 7725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:45.889011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.3:33577 (local address 127.3.18.1:35891)
0928 20:19:45.889234 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:45.889242 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:45.889325 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:45.889482 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:45.889483 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:45.889484 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:19:45.889534 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:45.889535 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:45.890411 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:45.890412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:45.891808 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:45.891813 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:45.892626 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:45.892627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:45.892726 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:45.892733 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 20:19:45.892803 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 20:19:45.892854 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":31,"thread_start_us":122,"threads_started":1}
I20260928 20:19:45.893649 7520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78821955484a4838be8af47487dbe815" candidate_uuid: "d224f54e51a24227a6b7a34e87d8e0b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d46718e3c33c4be79d574901e48a5eeb" is_pre_election: true
I20260928 20:19:45.893675 7653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78821955484a4838be8af47487dbe815" candidate_uuid: "d224f54e51a24227a6b7a34e87d8e0b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c8538aa487949c6905f36a294448ce4" is_pre_election: true
I20260928 20:19:45.893823 7520 raft_consensus.cc:2454] T 78821955484a4838be8af47487dbe815 P d46718e3c33c4be79d574901e48a5eeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d224f54e51a24227a6b7a34e87d8e0b0 in term 0.
I20260928 20:19:45.893839 7653 raft_consensus.cc:2454] T 78821955484a4838be8af47487dbe815 P 8c8538aa487949c6905f36a294448ce4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d224f54e51a24227a6b7a34e87d8e0b0 in term 0.
I20260928 20:19:45.894094 7323 leader_election.cc:302] T 78821955484a4838be8af47487dbe815 P d224f54e51a24227a6b7a34e87d8e0b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c8538aa487949c6905f36a294448ce4, d224f54e51a24227a6b7a34e87d8e0b0; no voters:
I20260928 20:19:45.894253 7723 raft_consensus.cc:2781] T 78821955484a4838be8af47487dbe815 P d224f54e51a24227a6b7a34e87d8e0b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 20:19:45.894315 7723 raft_consensus.cc:482] T 78821955484a4838be8af47487dbe815 P d224f54e51a24227a6b7a34e87d8e0b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 20:19:45.894332 7723 raft_consensus.cc:3037] T 78821955484a4838be8af47487dbe815 P d224f54e51a24227a6b7a34e87d8e0b0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:45.895013 7723 raft_consensus.cc:504] T 78821955484a4838be8af47487dbe815 P d224f54e51a24227a6b7a34e87d8e0b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d224f54e51a24227a6b7a34e87d8e0b0" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34125 } } peers { permanent_uuid: "8c8538aa487949c6905f36a294448ce4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 33577 } } peers { permanent_uuid: "d46718e3c33c4be79d574901e48a5eeb" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 37435 } }
I20260928 20:19:45.895169 7723 leader_election.cc:288] T 78821955484a4838be8af47487dbe815 P d224f54e51a24227a6b7a34e87d8e0b0 [CANDIDATE]: Term 1 election: Requested vote from peers 8c8538aa487949c6905f36a294448ce4 (127.3.18.3:33577), d46718e3c33c4be79d574901e48a5eeb (127.3.18.2:37435)
I20260928 20:19:45.895344 7653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78821955484a4838be8af47487dbe815" candidate_uuid: "d224f54e51a24227a6b7a34e87d8e0b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c8538aa487949c6905f36a294448ce4"
I20260928 20:19:45.895377 7520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78821955484a4838be8af47487dbe815" candidate_uuid: "d224f54e51a24227a6b7a34e87d8e0b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d46718e3c33c4be79d574901e48a5eeb"
I20260928 20:19:45.895434 7653 raft_consensus.cc:3037] T 78821955484a4838be8af47487dbe815 P 8c8538aa487949c6905f36a294448ce4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:45.895473 7520 raft_consensus.cc:3037] T 78821955484a4838be8af47487dbe815 P d46718e3c33c4be79d574901e48a5eeb [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:45.896229 7653 raft_consensus.cc:2454] T 78821955484a4838be8af47487dbe815 P 8c8538aa487949c6905f36a294448ce4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d224f54e51a24227a6b7a34e87d8e0b0 in term 1.
I20260928 20:19:45.896366 7520 raft_consensus.cc:2454] T 78821955484a4838be8af47487dbe815 P d46718e3c33c4be79d574901e48a5eeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d224f54e51a24227a6b7a34e87d8e0b0 in term 1.
I20260928 20:19:45.896415 7323 leader_election.cc:302] T 78821955484a4838be8af47487dbe815 P d224f54e51a24227a6b7a34e87d8e0b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c8538aa487949c6905f36a294448ce4, d224f54e51a24227a6b7a34e87d8e0b0; no voters:
I20260928 20:19:45.896498 7723 raft_consensus.cc:2781] T 78821955484a4838be8af47487dbe815 P d224f54e51a24227a6b7a34e87d8e0b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:19:45.896662 7723 raft_consensus.cc:686] T 78821955484a4838be8af47487dbe815 P d224f54e51a24227a6b7a34e87d8e0b0 [term 1 LEADER]: Becoming Leader. State: Replica: d224f54e51a24227a6b7a34e87d8e0b0, State: Running, Role: LEADER
I20260928 20:19:45.896765 7723 consensus_queue.cc:237] T 78821955484a4838be8af47487dbe815 P d224f54e51a24227a6b7a34e87d8e0b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d224f54e51a24227a6b7a34e87d8e0b0" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34125 } } peers { permanent_uuid: "8c8538aa487949c6905f36a294448ce4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 33577 } } peers { permanent_uuid: "d46718e3c33c4be79d574901e48a5eeb" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 37435 } }
I20260928 20:19:45.897701 7243 catalog_manager.cc:5495] T 78821955484a4838be8af47487dbe815 P d224f54e51a24227a6b7a34e87d8e0b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to d224f54e51a24227a6b7a34e87d8e0b0 (127.3.18.1). New cstate: current_term: 1 leader_uuid: "d224f54e51a24227a6b7a34e87d8e0b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d224f54e51a24227a6b7a34e87d8e0b0" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c8538aa487949c6905f36a294448ce4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 33577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d46718e3c33c4be79d574901e48a5eeb" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 37435 } health_report { overall_health: UNKNOWN } } }
I20260928 20:19:45.914911 7715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:45.912274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36950 (local address 127.3.18.1:34125)
0928 20:19:45.912343 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:45.912346 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:45.912356 (+ 10us) server_negotiation.cc:407] Connection header received
0928 20:19:45.912502 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:45.912503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:45.912525 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:45.912606 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:19:45.912761 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:45.913229 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:45.914021 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:45.914213 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:45.914287 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:45.914511 (+ 224us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:19:45.914618 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:19:45.914749 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:45.914767 (+ 18us) server_negotiation.cc:299] Negotiation successful
0928 20:19:45.914790 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 20:19:45.916857 7653 raft_consensus.cc:1261] T 78821955484a4838be8af47487dbe815 P 8c8538aa487949c6905f36a294448ce4 [term 1 FOLLOWER]: Refusing update from remote peer d224f54e51a24227a6b7a34e87d8e0b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:19:45.916920 7520 raft_consensus.cc:1261] T 78821955484a4838be8af47487dbe815 P d46718e3c33c4be79d574901e48a5eeb [term 1 FOLLOWER]: Refusing update from remote peer d224f54e51a24227a6b7a34e87d8e0b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:19:45.917320 7726 consensus_queue.cc:1035] T 78821955484a4838be8af47487dbe815 P d224f54e51a24227a6b7a34e87d8e0b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c8538aa487949c6905f36a294448ce4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 33577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:19:45.917472 7723 consensus_queue.cc:1035] T 78821955484a4838be8af47487dbe815 P d224f54e51a24227a6b7a34e87d8e0b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d46718e3c33c4be79d574901e48a5eeb" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 37435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:19:45.919806 7729 mvcc.cc:205] Tried to move back new op lower bound from 7334407315112427520 to 7334407315033915392. Current Snapshot: MvccSnapshot[applied={T|T < 7334407315112427520}]
I20260928 20:19:45.920586 7731 mvcc.cc:205] Tried to move back new op lower bound from 7334407315112427520 to 7334407315033915392. Current Snapshot: MvccSnapshot[applied={T|T < 7334407315112427520}]
I20260928 20:19:45.920779 7730 mvcc.cc:205] Tried to move back new op lower bound from 7334407315112427520 to 7334407315033915392. Current Snapshot: MvccSnapshot[applied={T|T < 7334407315112427520}]
I20260928 20:19:45.922228 7241 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:19:45.922334 7241 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:19:45.923252 7241 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 88e9b658fefb43dbbd2406d81beb5930: Sending DeleteTablet for 3 replicas of tablet 78821955484a4838be8af47487dbe815
I20260928 20:19:45.923641 7367 tablet_service.cc:1514] Processing DeleteTablet for tablet 78821955484a4838be8af47487dbe815 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:19:45 UTC) from {username='kudu', principal='kudu/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:36948
I20260928 20:19:45.923692 7633 tablet_service.cc:1514] Processing DeleteTablet for tablet 78821955484a4838be8af47487dbe815 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:19:45 UTC) from {username='kudu', principal='kudu/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:53476
I20260928 20:19:45.923867 7500 tablet_service.cc:1514] Processing DeleteTablet for tablet 78821955484a4838be8af47487dbe815 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:19:45 UTC) from {username='kudu', principal='kudu/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:49380
I20260928 20:19:45.924129 7740 tablet_replica.cc:330] T 78821955484a4838be8af47487dbe815 P d46718e3c33c4be79d574901e48a5eeb: stopping tablet replica
I20260928 20:19:45.924276 7740 raft_consensus.cc:2229] T 78821955484a4838be8af47487dbe815 P d46718e3c33c4be79d574901e48a5eeb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:19:45.924271 7741 tablet_replica.cc:330] T 78821955484a4838be8af47487dbe815 P d224f54e51a24227a6b7a34e87d8e0b0: stopping tablet replica
I20260928 20:19:45.924376 7741 raft_consensus.cc:2229] T 78821955484a4838be8af47487dbe815 P d224f54e51a24227a6b7a34e87d8e0b0 [term 1 LEADER]: Raft consensus shutting down.
I20260928 20:19:45.924430 7740 raft_consensus.cc:2258] T 78821955484a4838be8af47487dbe815 P d46718e3c33c4be79d574901e48a5eeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:19:45.924510 7741 raft_consensus.cc:2258] T 78821955484a4838be8af47487dbe815 P d224f54e51a24227a6b7a34e87d8e0b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:19:45.924510 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 7305
I20260928 20:19:45.925669 7739 tablet_replica.cc:330] T 78821955484a4838be8af47487dbe815 P 8c8538aa487949c6905f36a294448ce4: stopping tablet replica
I20260928 20:19:45.925784 7740 ts_tablet_manager.cc:1905] T 78821955484a4838be8af47487dbe815 P d46718e3c33c4be79d574901e48a5eeb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:19:45.925796 7739 raft_consensus.cc:2229] T 78821955484a4838be8af47487dbe815 P 8c8538aa487949c6905f36a294448ce4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:19:45.925904 7739 raft_consensus.cc:2258] T 78821955484a4838be8af47487dbe815 P 8c8538aa487949c6905f36a294448ce4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:19:45.926362 7739 ts_tablet_manager.cc:1905] T 78821955484a4838be8af47487dbe815 P 8c8538aa487949c6905f36a294448ce4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:19:45.928006 7739 ts_tablet_manager.cc:1918] T 78821955484a4838be8af47487dbe815 P 8c8538aa487949c6905f36a294448ce4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:19:45.928107 7739 log.cc:1196] T 78821955484a4838be8af47487dbe815 P 8c8538aa487949c6905f36a294448ce4: Deleting WAL directory at /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/minicluster-data/ts-2/wal/wals/78821955484a4838be8af47487dbe815
I20260928 20:19:45.928097 7740 ts_tablet_manager.cc:1918] T 78821955484a4838be8af47487dbe815 P d46718e3c33c4be79d574901e48a5eeb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:19:45.928169 7740 log.cc:1196] T 78821955484a4838be8af47487dbe815 P d46718e3c33c4be79d574901e48a5eeb: Deleting WAL directory at /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790626764250411-3144-0/minicluster-data/ts-1/wal/wals/78821955484a4838be8af47487dbe815
I20260928 20:19:45.928426 7739 ts_tablet_manager.cc:1939] T 78821955484a4838be8af47487dbe815 P 8c8538aa487949c6905f36a294448ce4: Deleting consensus metadata
I20260928 20:19:45.928454 7740 ts_tablet_manager.cc:1939] T 78821955484a4838be8af47487dbe815 P d46718e3c33c4be79d574901e48a5eeb: Deleting consensus metadata
I20260928 20:19:45.928936 7230 catalog_manager.cc:4841] TS 8c8538aa487949c6905f36a294448ce4 (127.3.18.3:33577): tablet 78821955484a4838be8af47487dbe815 (table test-table [id=abd2cfe6d6164d1bbe870f7e47a9fd5f]) successfully deleted
I20260928 20:19:45.928937 7229 catalog_manager.cc:4841] TS d46718e3c33c4be79d574901e48a5eeb (127.3.18.2:37435): tablet 78821955484a4838be8af47487dbe815 (table test-table [id=abd2cfe6d6164d1bbe870f7e47a9fd5f]) successfully deleted
W20260928 20:19:45.930843 7228 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.18.1:34125 (error 108)
I20260928 20:19:45.931056 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 7438
I20260928 20:19:45.932866 7710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:45.932661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.1:34125 (local address 127.0.0.1:36956)
0928 20:19:45.932741 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:45.932796 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.1:34125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 20:19:45.933020 7228 catalog_manager.cc:4584] TS d224f54e51a24227a6b7a34e87d8e0b0 (127.3.18.1:34125): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 78821955484a4838be8af47487dbe815: Network error: Client connection negotiation failed: client connection to 127.3.18.1:34125: connect: Connection refused (error 111)
I20260928 20:19:45.936021 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 7571
I20260928 20:19:45.940862 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 7212
2026-09-28T20:19:45Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2000 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T20:19:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:19:45Z Disabled control of system clock
I20260928 20:19:45.970047 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790626764250411-3144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790626764250411-3144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790626764250411-3144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:40821
--webserver_interface=127.3.18.62
--webserver_port=0
--builtin_ntp_servers=127.3.18.20:42209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.18.62:40821
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 20:19:46.055675 7746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:19:46.055945 7746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:19:46.055970 7746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:19:46.057173 7746 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:19:46.057217 7746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:19:46.057230 7746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:19:46.057245 7746 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:19:46.057256 7746 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:19:46.058653 7746 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:42209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790626764250411-3144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790626764250411-3144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.18.62:40821
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:40821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.18.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790626764250411-3144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:19:46.058872 7746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:19:46.059135 7746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:19:46.064715 7754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:46.064710 7752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:46.064765 7746 server_base.cc:1034] running on GCE node
W20260928 20:19:46.064797 7751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:46.065240 7746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:19:46.065654 7746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:19:46.066846 7746 hybrid_clock.cc:648] HybridClock initialized: now 1790626786066806 us; error 73 us; skew 500 ppm
I20260928 20:19:46.068591 7746 webserver.cc:460] Webserver started at http://127.3.18.62:46369/ using document root <none> and password file <none>
I20260928 20:19:46.068864 7746 fs_manager.cc:359] Metadata directory not provided
I20260928 20:19:46.068936 7746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:19:46.069094 7746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:19:46.070236 7746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790626764250411-3144-0/minicluster-data/master-0/data/instance:
uuid: "c08c01564b1d44bc9be8d30f150daf69"
format_stamp: "Formatted at 2026-09-28 20:19:46 on dist-test-slave-b83v"
server_key: "090e7110f0cb944e19fec0300be9179c"
server_key_iv: "cf34a3f66463960a5814babcbf1dba9f"
server_key_version: "encryptionkey@0"
I20260928 20:19:46.070628 7746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790626764250411-3144-0/minicluster-data/master-0/wal/instance:
uuid: "c08c01564b1d44bc9be8d30f150daf69"
format_stamp: "Formatted at 2026-09-28 20:19:46 on dist-test-slave-b83v"
server_key: "090e7110f0cb944e19fec0300be9179c"
server_key_iv: "cf34a3f66463960a5814babcbf1dba9f"
server_key_version: "encryptionkey@0"
I20260928 20:19:46.072088 7746 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260928 20:19:46.072918 7760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:46.073089 7746 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 20:19:46.073192 7746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790626764250411-3144-0/minicluster-data/master-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790626764250411-3144-0/minicluster-data/master-0/wal
uuid: "c08c01564b1d44bc9be8d30f150daf69"
format_stamp: "Formatted at 2026-09-28 20:19:46 on dist-test-slave-b83v"
server_key: "090e7110f0cb944e19fec0300be9179c"
server_key_iv: "cf34a3f66463960a5814babcbf1dba9f"
server_key_version: "encryptionkey@0"
I20260928 20:19:46.073268 7746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790626764250411-3144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790626764250411-3144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790626764250411-3144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:19:46.077003 7746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:19:46.077283 7746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:19:46.077435 7746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:19:46.082070 7746 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.62:40821
I20260928 20:19:46.082098 7812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.62:40821 every 8 connection(s)
I20260928 20:19:46.082461 7746 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
I20260928 20:19:46.083168 7813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:46.085247 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 7746
I20260928 20:19:46.085386 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790626764250411-3144-0/minicluster-data/master-0/wal/instance
I20260928 20:19:46.085556 3144 external_mini_cluster.cc:1428] Setting key 23245b3adae1be6433d4ea1a21c33db6
I20260928 20:19:46.086457 7813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c08c01564b1d44bc9be8d30f150daf69: Bootstrap starting.
I20260928 20:19:46.087087 7813 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c08c01564b1d44bc9be8d30f150daf69: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:46.087459 7813 log.cc:824] T 00000000000000000000000000000000 P c08c01564b1d44bc9be8d30f150daf69: Log is configured to *not* fsync() on all Append() calls
I20260928 20:19:46.088459 7813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c08c01564b1d44bc9be8d30f150daf69: No bootstrap required, opened a new log
I20260928 20:19:46.088481 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:46.086218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58764 (local address 127.3.18.62:40821)
0928 20:19:46.086465 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:46.086473 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:46.086499 (+ 26us) server_negotiation.cc:407] Connection header received
0928 20:19:46.086971 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:46.086978 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:46.087022 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:46.087181 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:46.087966 (+ 785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:46.087970 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:46.087971 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:19:46.087975 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:46.088028 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:19:46.088043 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:46.088143 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:46.088237 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 20:19:46.088287 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":140,"threads_started":1}
I20260928 20:19:46.089856 7813 raft_consensus.cc:348] T 00000000000000000000000000000000 P c08c01564b1d44bc9be8d30f150daf69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c08c01564b1d44bc9be8d30f150daf69" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 40821 } }
I20260928 20:19:46.090026 7813 raft_consensus.cc:374] T 00000000000000000000000000000000 P c08c01564b1d44bc9be8d30f150daf69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:46.090073 7813 raft_consensus.cc:729] T 00000000000000000000000000000000 P c08c01564b1d44bc9be8d30f150daf69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c08c01564b1d44bc9be8d30f150daf69, State: Initialized, Role: FOLLOWER
I20260928 20:19:46.090199 7813 consensus_queue.cc:260] T 00000000000000000000000000000000 P c08c01564b1d44bc9be8d30f150daf69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c08c01564b1d44bc9be8d30f150daf69" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 40821 } }
I20260928 20:19:46.090301 7813 raft_consensus.cc:388] T 00000000000000000000000000000000 P c08c01564b1d44bc9be8d30f150daf69 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:19:46.090344 7813 raft_consensus.cc:482] T 00000000000000000000000000000000 P c08c01564b1d44bc9be8d30f150daf69 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:19:46.090392 7813 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c08c01564b1d44bc9be8d30f150daf69 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:46.091251 7813 raft_consensus.cc:504] T 00000000000000000000000000000000 P c08c01564b1d44bc9be8d30f150daf69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c08c01564b1d44bc9be8d30f150daf69" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 40821 } }
I20260928 20:19:46.091379 7813 leader_election.cc:302] T 00000000000000000000000000000000 P c08c01564b1d44bc9be8d30f150daf69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c08c01564b1d44bc9be8d30f150daf69; no voters:
I20260928 20:19:46.091544 7813 leader_election.cc:288] T 00000000000000000000000000000000 P c08c01564b1d44bc9be8d30f150daf69 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:19:46.091584 7818 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c08c01564b1d44bc9be8d30f150daf69 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:19:46.091784 7818 raft_consensus.cc:686] T 00000000000000000000000000000000 P c08c01564b1d44bc9be8d30f150daf69 [term 1 LEADER]: Becoming Leader. State: Replica: c08c01564b1d44bc9be8d30f150daf69, State: Running, Role: LEADER
I20260928 20:19:46.091837 7813 sys_catalog.cc:564] T 00000000000000000000000000000000 P c08c01564b1d44bc9be8d30f150daf69 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:19:46.091890 7818 consensus_queue.cc:237] T 00000000000000000000000000000000 P c08c01564b1d44bc9be8d30f150daf69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c08c01564b1d44bc9be8d30f150daf69" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 40821 } }
I20260928 20:19:46.092303 7819 sys_catalog.cc:455] T 00000000000000000000000000000000 P c08c01564b1d44bc9be8d30f150daf69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c08c01564b1d44bc9be8d30f150daf69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c08c01564b1d44bc9be8d30f150daf69" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 40821 } } }
I20260928 20:19:46.092418 7819 sys_catalog.cc:458] T 00000000000000000000000000000000 P c08c01564b1d44bc9be8d30f150daf69 [sys.catalog]: This master's current role is: LEADER
I20260928 20:19:46.092597 7820 sys_catalog.cc:455] T 00000000000000000000000000000000 P c08c01564b1d44bc9be8d30f150daf69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c08c01564b1d44bc9be8d30f150daf69. Latest consensus state: current_term: 1 leader_uuid: "c08c01564b1d44bc9be8d30f150daf69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c08c01564b1d44bc9be8d30f150daf69" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 40821 } } }
I20260928 20:19:46.092619 7827 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:19:46.092679 7820 sys_catalog.cc:458] T 00000000000000000000000000000000 P c08c01564b1d44bc9be8d30f150daf69 [sys.catalog]: This master's current role is: LEADER
I20260928 20:19:46.093125 7827 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:19:46.094841 7827 catalog_manager.cc:1347] Generated new cluster ID: e5d4b431e7e84160b72247be54d6bb4c
I20260928 20:19:46.094918 7827 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:19:46.110747 7827 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:19:46.110958 7827 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:19:46.119392 7827 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c08c01564b1d44bc9be8d30f150daf69: Generated new TSK 0
I20260928 20:19:46.119633 7827 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 20:19:46.122329 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790626764250411-3144-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790626764250411-3144-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790626764250411-3144-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.1:0
--local_ip_for_outbound_sockets=127.3.18.1
--webserver_interface=127.3.18.1
--webserver_port=0
--tserver_master_addrs=127.3.18.62:40821
--builtin_ntp_servers=127.3.18.20:42209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 20:19:46.211813 7837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:19:46.212082 7837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:19:46.212118 7837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:19:46.213475 7837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:19:46.213522 7837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:19:46.213550 7837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.1
I20260928 20:19:46.215018 7837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:42209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790626764250411-3144-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790626764250411-3144-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.18.1
--webserver_port=0
--tserver_master_addrs=127.3.18.62:40821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7837
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.1
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790626764250411-3144-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:19:46.215317 7837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:19:46.215644 7837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:19:46.222632 7842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:46.222705 7843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:46.222694 7845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:46.223111 7837 server_base.cc:1034] running on GCE node
I20260928 20:19:46.223331 7837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:19:46.223644 7837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:19:46.224893 7837 hybrid_clock.cc:648] HybridClock initialized: now 1790626786224873 us; error 58 us; skew 500 ppm
I20260928 20:19:46.226454 7837 webserver.cc:460] Webserver started at http://127.3.18.1:35789/ using document root <none> and password file <none>
I20260928 20:19:46.226768 7837 fs_manager.cc:359] Metadata directory not provided
I20260928 20:19:46.226866 7837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:19:46.227053 7837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:19:46.228353 7837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790626764250411-3144-0/minicluster-data/ts-0/data/instance:
uuid: "9808c5d773f641e894a7df1d78643909"
format_stamp: "Formatted at 2026-09-28 20:19:46 on dist-test-slave-b83v"
server_key: "3c15ce665fa11a8757eb6e04474a0b60"
server_key_iv: "a4520783122a08b8b36c7eab3ebc6681"
server_key_version: "encryptionkey@0"
I20260928 20:19:46.228746 7837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790626764250411-3144-0/minicluster-data/ts-0/wal/instance:
uuid: "9808c5d773f641e894a7df1d78643909"
format_stamp: "Formatted at 2026-09-28 20:19:46 on dist-test-slave-b83v"
server_key: "3c15ce665fa11a8757eb6e04474a0b60"
server_key_iv: "a4520783122a08b8b36c7eab3ebc6681"
server_key_version: "encryptionkey@0"
I20260928 20:19:46.230216 7837 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 20:19:46.231107 7851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:46.231298 7837 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:19:46.231390 7837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790626764250411-3144-0/minicluster-data/ts-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790626764250411-3144-0/minicluster-data/ts-0/wal
uuid: "9808c5d773f641e894a7df1d78643909"
format_stamp: "Formatted at 2026-09-28 20:19:46 on dist-test-slave-b83v"
server_key: "3c15ce665fa11a8757eb6e04474a0b60"
server_key_iv: "a4520783122a08b8b36c7eab3ebc6681"
server_key_version: "encryptionkey@0"
I20260928 20:19:46.231521 7837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790626764250411-3144-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790626764250411-3144-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790626764250411-3144-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:19:46.235399 7837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:19:46.235706 7837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:19:46.235888 7837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:19:46.236126 7837 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:19:46.236464 7837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:19:46.236521 7837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:46.236562 7837 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:19:46.236588 7837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:46.242549 7837 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.1:33781
I20260928 20:19:46.242628 7964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.1:33781 every 8 connection(s)
I20260928 20:19:46.243034 7837 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
I20260928 20:19:46.246757 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:46.244080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:36533 (local address 127.3.18.62:40821)
0928 20:19:46.244156 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:46.244159 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:46.244329 (+ 170us) server_negotiation.cc:407] Connection header received
0928 20:19:46.244905 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:46.244906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:46.244913 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:46.244972 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:46.246469 (+ 1497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:46.246470 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:46.246471 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:19:46.246475 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:46.246493 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:19:46.246497 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:46.246583 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:46.246667 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 20:19:46.246683 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 20:19:46.247016 7967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:46.243944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40821 (local address 127.3.18.1:36533)
0928 20:19:46.244178 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:46.244224 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:46.244777 (+ 553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:46.245053 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:46.245058 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:46.245068 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:46.246244 (+ 1176us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 20:19:46.246245 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:46.246276 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 20:19:46.246358 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:46.246587 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:46.246593 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:46.246596 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 20:19:46.246671 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 20:19:46.246738 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":97,"threads_started":1}
I20260928 20:19:46.247648 7965 heartbeater.cc:347] Connected to a master server at 127.3.18.62:40821
I20260928 20:19:46.247750 7965 heartbeater.cc:464] Registering TS with master...
I20260928 20:19:46.247964 7965 heartbeater.cc:511] Master 127.3.18.62:40821 requested a full tablet report, sending...
I20260928 20:19:46.247946 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 7837
I20260928 20:19:46.248071 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790626764250411-3144-0/minicluster-data/ts-0/wal/instance
I20260928 20:19:46.248260 3144 external_mini_cluster.cc:1428] Setting key 163fe44c758b30ad7dc1442e6d60214a
I20260928 20:19:46.248569 7776 ts_manager.cc:194] Registered new tserver with Master: 9808c5d773f641e894a7df1d78643909 (127.3.18.1:33781)
I20260928 20:19:46.249289 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790626764250411-3144-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790626764250411-3144-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790626764250411-3144-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.2:0
--local_ip_for_outbound_sockets=127.3.18.2
--webserver_interface=127.3.18.2
--webserver_port=0
--tserver_master_addrs=127.3.18.62:40821
--builtin_ntp_servers=127.3.18.20:42209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 20:19:46.331657 7968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:19:46.332020 7968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:19:46.332062 7968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:19:46.333390 7968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:19:46.333475 7968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:19:46.333523 7968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.2
I20260928 20:19:46.335111 7968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:42209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790626764250411-3144-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790626764250411-3144-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.18.2
--webserver_port=0
--tserver_master_addrs=127.3.18.62:40821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.2
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790626764250411-3144-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:19:46.335414 7968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:19:46.335866 7968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 20:19:46.341346 7968 server_base.cc:1034] running on GCE node
W20260928 20:19:46.341327 7976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:46.341305 7973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:46.341327 7974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:46.341910 7968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:19:46.342211 7968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:19:46.343413 7968 hybrid_clock.cc:648] HybridClock initialized: now 1790626786343381 us; error 49 us; skew 500 ppm
I20260928 20:19:46.345126 7968 webserver.cc:460] Webserver started at http://127.3.18.2:36581/ using document root <none> and password file <none>
I20260928 20:19:46.345431 7968 fs_manager.cc:359] Metadata directory not provided
I20260928 20:19:46.345516 7968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:19:46.345681 7968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:19:46.346939 7968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790626764250411-3144-0/minicluster-data/ts-1/data/instance:
uuid: "b65a673f17f8487f8e9431585a7cf0d0"
format_stamp: "Formatted at 2026-09-28 20:19:46 on dist-test-slave-b83v"
server_key: "a4e8e11af385ff2a1f2e9c9729221b9f"
server_key_iv: "d8ea8b13494ac2c1b5b400fca5e4e10d"
server_key_version: "encryptionkey@0"
I20260928 20:19:46.347373 7968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790626764250411-3144-0/minicluster-data/ts-1/wal/instance:
uuid: "b65a673f17f8487f8e9431585a7cf0d0"
format_stamp: "Formatted at 2026-09-28 20:19:46 on dist-test-slave-b83v"
server_key: "a4e8e11af385ff2a1f2e9c9729221b9f"
server_key_iv: "d8ea8b13494ac2c1b5b400fca5e4e10d"
server_key_version: "encryptionkey@0"
I20260928 20:19:46.349045 7968 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 20:19:46.350059 7982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:46.350257 7968 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:19:46.350410 7968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790626764250411-3144-0/minicluster-data/ts-1/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790626764250411-3144-0/minicluster-data/ts-1/wal
uuid: "b65a673f17f8487f8e9431585a7cf0d0"
format_stamp: "Formatted at 2026-09-28 20:19:46 on dist-test-slave-b83v"
server_key: "a4e8e11af385ff2a1f2e9c9729221b9f"
server_key_iv: "d8ea8b13494ac2c1b5b400fca5e4e10d"
server_key_version: "encryptionkey@0"
I20260928 20:19:46.350517 7968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790626764250411-3144-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790626764250411-3144-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790626764250411-3144-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:19:46.355104 7968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:19:46.355435 7968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:19:46.355602 7968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:19:46.355906 7968 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:19:46.356261 7968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:19:46.356321 7968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:46.356365 7968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:19:46.356400 7968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:46.362792 7968 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.2:43121
I20260928 20:19:46.362872 8095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.2:43121 every 8 connection(s)
I20260928 20:19:46.363276 7968 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
I20260928 20:19:46.364543 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 7968
I20260928 20:19:46.364641 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790626764250411-3144-0/minicluster-data/ts-1/wal/instance
I20260928 20:19:46.364840 3144 external_mini_cluster.cc:1428] Setting key 8ec2cb30d9afd5003504b6bd030831b5
I20260928 20:19:46.366070 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790626764250411-3144-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790626764250411-3144-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790626764250411-3144-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.3:0
--local_ip_for_outbound_sockets=127.3.18.3
--webserver_interface=127.3.18.3
--webserver_port=0
--tserver_master_addrs=127.3.18.62:40821
--builtin_ntp_servers=127.3.18.20:42209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 20:19:46.367276 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:46.364213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:44281 (local address 127.3.18.62:40821)
0928 20:19:46.364279 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:46.364281 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:46.364540 (+ 259us) server_negotiation.cc:407] Connection header received
0928 20:19:46.365246 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:46.365247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:46.365254 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:46.365342 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:46.366888 (+ 1546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:46.366890 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:46.366890 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:19:46.366896 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:46.366915 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:19:46.366922 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:46.367000 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:46.367151 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 20:19:46.367169 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 20:19:46.367511 8098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:46.364179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40821 (local address 127.3.18.2:44281)
0928 20:19:46.364382 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:46.364429 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:46.365136 (+ 707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:46.365421 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:46.365425 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:46.365434 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:46.366689 (+ 1255us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 20:19:46.366690 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:46.366724 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 20:19:46.366758 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:46.367020 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:46.367028 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:46.367032 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:19:46.367133 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 20:19:46.367215 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":109,"threads_started":1}
I20260928 20:19:46.368125 8096 heartbeater.cc:347] Connected to a master server at 127.3.18.62:40821
I20260928 20:19:46.368249 8096 heartbeater.cc:464] Registering TS with master...
I20260928 20:19:46.368499 8096 heartbeater.cc:511] Master 127.3.18.62:40821 requested a full tablet report, sending...
I20260928 20:19:46.368919 7777 ts_manager.cc:194] Registered new tserver with Master: b65a673f17f8487f8e9431585a7cf0d0 (127.3.18.2:43121)
W20260928 20:19:46.450616 8099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:19:46.450841 8099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:19:46.450865 8099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:19:46.452378 8099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:19:46.452435 8099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:19:46.452463 8099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.3
I20260928 20:19:46.453909 8099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:42209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790626764250411-3144-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790626764250411-3144-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.18.3
--webserver_port=0
--tserver_master_addrs=127.3.18.62:40821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.3
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790626764250411-3144-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:19:46.454200 8099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:19:46.454527 8099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:19:46.459973 8104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:46.460022 8107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:46.460108 8099 server_base.cc:1034] running on GCE node
W20260928 20:19:46.459973 8105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:46.460498 8099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:19:46.460870 8099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:19:46.462031 8099 hybrid_clock.cc:648] HybridClock initialized: now 1790626786462018 us; error 28 us; skew 500 ppm
I20260928 20:19:46.463361 8099 webserver.cc:460] Webserver started at http://127.3.18.3:46827/ using document root <none> and password file <none>
I20260928 20:19:46.463693 8099 fs_manager.cc:359] Metadata directory not provided
I20260928 20:19:46.463745 8099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:19:46.463938 8099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:19:46.465139 8099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790626764250411-3144-0/minicluster-data/ts-2/data/instance:
uuid: "19c23e487a9a4a5b81937750076082ca"
format_stamp: "Formatted at 2026-09-28 20:19:46 on dist-test-slave-b83v"
server_key: "4f6295fe313266bc9348d30becdb676c"
server_key_iv: "175df3d4aa0fcd81fdb3ccd60b9578a6"
server_key_version: "encryptionkey@0"
I20260928 20:19:46.465528 8099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790626764250411-3144-0/minicluster-data/ts-2/wal/instance:
uuid: "19c23e487a9a4a5b81937750076082ca"
format_stamp: "Formatted at 2026-09-28 20:19:46 on dist-test-slave-b83v"
server_key: "4f6295fe313266bc9348d30becdb676c"
server_key_iv: "175df3d4aa0fcd81fdb3ccd60b9578a6"
server_key_version: "encryptionkey@0"
I20260928 20:19:46.466923 8099 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:19:46.468024 8113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:46.468236 8099 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:19:46.468350 8099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790626764250411-3144-0/minicluster-data/ts-2/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790626764250411-3144-0/minicluster-data/ts-2/wal
uuid: "19c23e487a9a4a5b81937750076082ca"
format_stamp: "Formatted at 2026-09-28 20:19:46 on dist-test-slave-b83v"
server_key: "4f6295fe313266bc9348d30becdb676c"
server_key_iv: "175df3d4aa0fcd81fdb3ccd60b9578a6"
server_key_version: "encryptionkey@0"
I20260928 20:19:46.468408 8099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790626764250411-3144-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790626764250411-3144-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790626764250411-3144-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:19:46.471864 8099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:19:46.472095 8099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:19:46.472187 8099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:19:46.472334 8099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:19:46.472592 8099 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:19:46.472621 8099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:46.472640 8099 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:19:46.472653 8099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:46.478264 8099 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.3:46405
I20260928 20:19:46.478346 8226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.3:46405 every 8 connection(s)
I20260928 20:19:46.478721 8099 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
I20260928 20:19:46.481658 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 8099
I20260928 20:19:46.481755 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790626764250411-3144-0/minicluster-data/ts-2/wal/instance
I20260928 20:19:46.482028 3144 external_mini_cluster.cc:1428] Setting key 6548bfd41b184c96b962f921c6f14d46
I20260928 20:19:46.482401 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:46.479649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:49677 (local address 127.3.18.62:40821)
0928 20:19:46.479748 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:46.479752 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:46.480080 (+ 328us) server_negotiation.cc:407] Connection header received
0928 20:19:46.480648 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:46.480649 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:46.480656 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:46.480722 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:46.482092 (+ 1370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:46.482093 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:46.482093 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:19:46.482099 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:46.482115 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:19:46.482119 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:46.482173 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:46.482287 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 20:19:46.482304 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 20:19:46.482728 8229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:46.479611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40821 (local address 127.3.18.3:49677)
0928 20:19:46.479926 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:46.479962 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:46.480530 (+ 568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:46.480778 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:46.480782 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:46.480790 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:46.481839 (+ 1049us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 20:19:46.481840 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:46.481951 (+ 111us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 20:19:46.481997 (+ 46us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:46.482180 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:46.482188 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:46.482192 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:19:46.482282 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 20:19:46.482426 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":126,"threads_started":1}
I20260928 20:19:46.483218 8227 heartbeater.cc:347] Connected to a master server at 127.3.18.62:40821
I20260928 20:19:46.483325 8227 heartbeater.cc:464] Registering TS with master...
I20260928 20:19:46.483500 8227 heartbeater.cc:511] Master 127.3.18.62:40821 requested a full tablet report, sending...
I20260928 20:19:46.483918 7777 ts_manager.cc:194] Registered new tserver with Master: 19c23e487a9a4a5b81937750076082ca (127.3.18.3:46405)
I20260928 20:19:46.494145 3144 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 20:19:46.496666 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:46.495611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58772 (local address 127.3.18.62:40821)
0928 20:19:46.495670 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:46.495673 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:46.495794 (+ 121us) server_negotiation.cc:407] Connection header received
0928 20:19:46.495894 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:46.495895 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:46.495900 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:46.495953 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:46.496493 (+ 540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:46.496494 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:46.496495 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:19:46.496498 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:46.496510 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:19:46.496513 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:46.496563 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:46.496598 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 20:19:46.496610 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 20:19:46.498302 7777 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58772:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 20:19:46.499985 7777 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 20:19:46.504144 8235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:46.502377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.2:43121 (local address 127.0.0.1:41490)
0928 20:19:46.502596 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:46.502640 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:46.502698 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:46.503256 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:46.503257 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:46.503261 (+ 4us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:46.503646 (+ 385us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 20:19:46.503647 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:46.503663 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 20:19:46.503691 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:46.503964 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:46.503970 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:46.503973 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 20:19:46.504041 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 20:19:46.504055 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":109,"threads_started":1}
I20260928 20:19:46.504189 8237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:46.502474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41490 (local address 127.3.18.2:43121)
0928 20:19:46.502810 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:46.502814 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:46.502831 (+ 17us) server_negotiation.cc:407] Connection header received
0928 20:19:46.502851 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:46.502853 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:46.502859 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:46.502952 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:46.503794 (+ 842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:46.503803 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:46.503804 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:19:46.503808 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:46.503847 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:19:46.503872 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:46.503939 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:46.504074 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 20:19:46.504093 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":71,"threads_started":1}
I20260928 20:19:46.504302 8238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:46.502541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43796 (local address 127.3.18.1:33781)
0928 20:19:46.502757 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:46.502762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:46.502773 (+ 11us) server_negotiation.cc:407] Connection header received
0928 20:19:46.502799 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:46.502801 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:46.502810 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:46.502897 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:46.503957 (+ 1060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:46.503965 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:46.503965 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:19:46.503969 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:46.504001 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:19:46.504023 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:46.504072 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:46.504198 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 20:19:46.504212 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":92,"threads_started":1}
I20260928 20:19:46.504706 8236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:46.502416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.1:33781 (local address 127.0.0.1:43796)
0928 20:19:46.502619 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:46.502640 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:46.502697 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:46.502909 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:46.502914 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:46.502921 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:46.503856 (+ 935us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 20:19:46.503856 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:46.503862 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 20:19:46.503873 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:46.504094 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:46.504094 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:46.504095 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 20:19:46.504133 (+ 38us) client_negotiation.cc:240] Negotiation successful
0928 20:19:46.504143 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":99,"threads_started":1}
I20260928 20:19:46.504755 8240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:46.502806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46612 (local address 127.3.18.3:46405)
0928 20:19:46.503162 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:46.503165 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:46.503280 (+ 115us) server_negotiation.cc:407] Connection header received
0928 20:19:46.503342 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:46.503345 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:46.503353 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:46.503424 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:46.504441 (+ 1017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:46.504450 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:46.504450 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:19:46.504456 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:46.504486 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:19:46.504501 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:46.504546 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:46.504638 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 20:19:46.504655 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":114,"threads_started":1}
I20260928 20:19:46.504710 8239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:46.502626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.3:46405 (local address 127.0.0.1:46612)
0928 20:19:46.503222 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:46.503232 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:46.503282 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:46.503716 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:46.503716 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:46.503719 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:46.504333 (+ 614us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 20:19:46.504334 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:46.504340 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 20:19:46.504351 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:46.504565 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:46.504566 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:46.504566 (+ 0us) client_negotiation.cc:769] Sending connection context
0928 20:19:46.504623 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 20:19:46.504635 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":54,"threads_started":1}
I20260928 20:19:46.505414 8161 tablet_service.cc:1467] Processing CreateTablet for tablet 85cae9437e224b4a868c6384caf818b4 (DEFAULT_TABLE table=test-table [id=f5aaf9a01ae4400cba9b40e0c2b9e281]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:19:46.505394 8030 tablet_service.cc:1467] Processing CreateTablet for tablet 85cae9437e224b4a868c6384caf818b4 (DEFAULT_TABLE table=test-table [id=f5aaf9a01ae4400cba9b40e0c2b9e281]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:19:46.505474 7899 tablet_service.cc:1467] Processing CreateTablet for tablet 85cae9437e224b4a868c6384caf818b4 (DEFAULT_TABLE table=test-table [id=f5aaf9a01ae4400cba9b40e0c2b9e281]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:19:46.505754 8161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85cae9437e224b4a868c6384caf818b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:46.505754 8030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85cae9437e224b4a868c6384caf818b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:46.505776 7899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85cae9437e224b4a868c6384caf818b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:46.508805 8241 tablet_bootstrap.cc:492] T 85cae9437e224b4a868c6384caf818b4 P 9808c5d773f641e894a7df1d78643909: Bootstrap starting.
I20260928 20:19:46.509379 8242 tablet_bootstrap.cc:492] T 85cae9437e224b4a868c6384caf818b4 P 19c23e487a9a4a5b81937750076082ca: Bootstrap starting.
I20260928 20:19:46.509649 8241 tablet_bootstrap.cc:654] T 85cae9437e224b4a868c6384caf818b4 P 9808c5d773f641e894a7df1d78643909: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:46.510023 8241 log.cc:824] T 85cae9437e224b4a868c6384caf818b4 P 9808c5d773f641e894a7df1d78643909: Log is configured to *not* fsync() on all Append() calls
I20260928 20:19:46.510070 8242 tablet_bootstrap.cc:654] T 85cae9437e224b4a868c6384caf818b4 P 19c23e487a9a4a5b81937750076082ca: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:46.510417 8242 log.cc:824] T 85cae9437e224b4a868c6384caf818b4 P 19c23e487a9a4a5b81937750076082ca: Log is configured to *not* fsync() on all Append() calls
I20260928 20:19:46.510792 8243 tablet_bootstrap.cc:492] T 85cae9437e224b4a868c6384caf818b4 P b65a673f17f8487f8e9431585a7cf0d0: Bootstrap starting.
I20260928 20:19:46.510983 8241 tablet_bootstrap.cc:492] T 85cae9437e224b4a868c6384caf818b4 P 9808c5d773f641e894a7df1d78643909: No bootstrap required, opened a new log
I20260928 20:19:46.511058 8241 ts_tablet_manager.cc:1397] T 85cae9437e224b4a868c6384caf818b4 P 9808c5d773f641e894a7df1d78643909: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 20:19:46.511304 8242 tablet_bootstrap.cc:492] T 85cae9437e224b4a868c6384caf818b4 P 19c23e487a9a4a5b81937750076082ca: No bootstrap required, opened a new log
I20260928 20:19:46.511400 8242 ts_tablet_manager.cc:1397] T 85cae9437e224b4a868c6384caf818b4 P 19c23e487a9a4a5b81937750076082ca: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 20:19:46.511478 8243 tablet_bootstrap.cc:654] T 85cae9437e224b4a868c6384caf818b4 P b65a673f17f8487f8e9431585a7cf0d0: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:46.511833 8243 log.cc:824] T 85cae9437e224b4a868c6384caf818b4 P b65a673f17f8487f8e9431585a7cf0d0: Log is configured to *not* fsync() on all Append() calls
I20260928 20:19:46.512542 8243 tablet_bootstrap.cc:492] T 85cae9437e224b4a868c6384caf818b4 P b65a673f17f8487f8e9431585a7cf0d0: No bootstrap required, opened a new log
I20260928 20:19:46.512605 8243 ts_tablet_manager.cc:1397] T 85cae9437e224b4a868c6384caf818b4 P b65a673f17f8487f8e9431585a7cf0d0: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 20:19:46.513763 8241 raft_consensus.cc:348] T 85cae9437e224b4a868c6384caf818b4 P 9808c5d773f641e894a7df1d78643909 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65a673f17f8487f8e9431585a7cf0d0" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 43121 } } peers { permanent_uuid: "19c23e487a9a4a5b81937750076082ca" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 46405 } } peers { permanent_uuid: "9808c5d773f641e894a7df1d78643909" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 33781 } }
I20260928 20:19:46.514035 8241 raft_consensus.cc:374] T 85cae9437e224b4a868c6384caf818b4 P 9808c5d773f641e894a7df1d78643909 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:46.514087 8241 raft_consensus.cc:729] T 85cae9437e224b4a868c6384caf818b4 P 9808c5d773f641e894a7df1d78643909 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9808c5d773f641e894a7df1d78643909, State: Initialized, Role: FOLLOWER
I20260928 20:19:46.514237 8241 consensus_queue.cc:260] T 85cae9437e224b4a868c6384caf818b4 P 9808c5d773f641e894a7df1d78643909 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65a673f17f8487f8e9431585a7cf0d0" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 43121 } } peers { permanent_uuid: "19c23e487a9a4a5b81937750076082ca" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 46405 } } peers { permanent_uuid: "9808c5d773f641e894a7df1d78643909" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 33781 } }
I20260928 20:19:46.514503 8241 ts_tablet_manager.cc:1428] T 85cae9437e224b4a868c6384caf818b4 P 9808c5d773f641e894a7df1d78643909: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 20:19:46.514617 7965 heartbeater.cc:503] Master 127.3.18.62:40821 was elected leader, sending a full tablet report...
I20260928 20:19:46.514428 8242 raft_consensus.cc:348] T 85cae9437e224b4a868c6384caf818b4 P 19c23e487a9a4a5b81937750076082ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65a673f17f8487f8e9431585a7cf0d0" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 43121 } } peers { permanent_uuid: "19c23e487a9a4a5b81937750076082ca" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 46405 } } peers { permanent_uuid: "9808c5d773f641e894a7df1d78643909" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 33781 } }
I20260928 20:19:46.514919 8242 raft_consensus.cc:374] T 85cae9437e224b4a868c6384caf818b4 P 19c23e487a9a4a5b81937750076082ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:46.514962 8242 raft_consensus.cc:729] T 85cae9437e224b4a868c6384caf818b4 P 19c23e487a9a4a5b81937750076082ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19c23e487a9a4a5b81937750076082ca, State: Initialized, Role: FOLLOWER
I20260928 20:19:46.514957 8243 raft_consensus.cc:348] T 85cae9437e224b4a868c6384caf818b4 P b65a673f17f8487f8e9431585a7cf0d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65a673f17f8487f8e9431585a7cf0d0" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 43121 } } peers { permanent_uuid: "19c23e487a9a4a5b81937750076082ca" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 46405 } } peers { permanent_uuid: "9808c5d773f641e894a7df1d78643909" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 33781 } }
I20260928 20:19:46.515087 8242 consensus_queue.cc:260] T 85cae9437e224b4a868c6384caf818b4 P 19c23e487a9a4a5b81937750076082ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65a673f17f8487f8e9431585a7cf0d0" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 43121 } } peers { permanent_uuid: "19c23e487a9a4a5b81937750076082ca" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 46405 } } peers { permanent_uuid: "9808c5d773f641e894a7df1d78643909" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 33781 } }
I20260928 20:19:46.515121 8243 raft_consensus.cc:374] T 85cae9437e224b4a868c6384caf818b4 P b65a673f17f8487f8e9431585a7cf0d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:46.515161 8243 raft_consensus.cc:729] T 85cae9437e224b4a868c6384caf818b4 P b65a673f17f8487f8e9431585a7cf0d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b65a673f17f8487f8e9431585a7cf0d0, State: Initialized, Role: FOLLOWER
I20260928 20:19:46.515365 8242 ts_tablet_manager.cc:1428] T 85cae9437e224b4a868c6384caf818b4 P 19c23e487a9a4a5b81937750076082ca: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260928 20:19:46.515373 8227 heartbeater.cc:503] Master 127.3.18.62:40821 was elected leader, sending a full tablet report...
I20260928 20:19:46.515308 8243 consensus_queue.cc:260] T 85cae9437e224b4a868c6384caf818b4 P b65a673f17f8487f8e9431585a7cf0d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65a673f17f8487f8e9431585a7cf0d0" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 43121 } } peers { permanent_uuid: "19c23e487a9a4a5b81937750076082ca" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 46405 } } peers { permanent_uuid: "9808c5d773f641e894a7df1d78643909" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 33781 } }
I20260928 20:19:46.515640 8243 ts_tablet_manager.cc:1428] T 85cae9437e224b4a868c6384caf818b4 P b65a673f17f8487f8e9431585a7cf0d0: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 20:19:46.515699 8096 heartbeater.cc:503] Master 127.3.18.62:40821 was elected leader, sending a full tablet report...
I20260928 20:19:46.521435 8248 raft_consensus.cc:482] T 85cae9437e224b4a868c6384caf818b4 P 19c23e487a9a4a5b81937750076082ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:19:46.521559 8248 raft_consensus.cc:504] T 85cae9437e224b4a868c6384caf818b4 P 19c23e487a9a4a5b81937750076082ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65a673f17f8487f8e9431585a7cf0d0" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 43121 } } peers { permanent_uuid: "19c23e487a9a4a5b81937750076082ca" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 46405 } } peers { permanent_uuid: "9808c5d773f641e894a7df1d78643909" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 33781 } }
I20260928 20:19:46.521873 8248 leader_election.cc:288] T 85cae9437e224b4a868c6384caf818b4 P 19c23e487a9a4a5b81937750076082ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b65a673f17f8487f8e9431585a7cf0d0 (127.3.18.2:43121), 9808c5d773f641e894a7df1d78643909 (127.3.18.1:33781)
I20260928 20:19:46.523231 8238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:46.522221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:56587 (local address 127.3.18.1:33781)
0928 20:19:46.522262 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:46.522265 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:46.522272 (+ 7us) server_negotiation.cc:407] Connection header received
0928 20:19:46.522298 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:46.522300 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:46.522305 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:46.522360 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:46.522986 (+ 626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:46.522987 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:46.522988 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:19:46.522992 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:46.523010 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:19:46.523014 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:46.523074 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:46.523132 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 20:19:46.523146 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 20:19:46.523228 8229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:46.522072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.1:33781 (local address 127.3.18.3:56587)
0928 20:19:46.522146 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:46.522159 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:46.522196 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:46.522379 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:46.522380 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:46.522383 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:46.522872 (+ 489us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 20:19:46.522904 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:46.522911 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 20:19:46.522923 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:46.523072 (+ 149us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:46.523073 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:46.523074 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 20:19:46.523126 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 20:19:46.523140 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260928 20:19:46.523377 8237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:46.522121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:46065 (local address 127.3.18.2:43121)
0928 20:19:46.522167 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:46.522168 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:46.522334 (+ 166us) server_negotiation.cc:407] Connection header received
0928 20:19:46.522436 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:46.522437 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:46.522443 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:46.522510 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:46.523142 (+ 632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:46.523143 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:46.523143 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:19:46.523147 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:46.523161 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:19:46.523165 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:46.523218 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:46.523284 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 20:19:46.523298 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 20:19:46.523406 8250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:46.522072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.2:43121 (local address 127.3.18.3:46065)
0928 20:19:46.522251 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:46.522262 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:46.522322 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:46.522519 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:46.522520 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:46.522522 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:46.523045 (+ 523us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 20:19:46.523046 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:46.523053 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 20:19:46.523064 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:46.523217 (+ 153us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:46.523217 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:46.523218 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 20:19:46.523286 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 20:19:46.523298 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":21,"thread_start_us":110,"threads_started":1}
I20260928 20:19:46.523669 7919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85cae9437e224b4a868c6384caf818b4" candidate_uuid: "19c23e487a9a4a5b81937750076082ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9808c5d773f641e894a7df1d78643909" is_pre_election: true
I20260928 20:19:46.523700 8050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85cae9437e224b4a868c6384caf818b4" candidate_uuid: "19c23e487a9a4a5b81937750076082ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b65a673f17f8487f8e9431585a7cf0d0" is_pre_election: true
I20260928 20:19:46.523838 7919 raft_consensus.cc:2454] T 85cae9437e224b4a868c6384caf818b4 P 9808c5d773f641e894a7df1d78643909 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19c23e487a9a4a5b81937750076082ca in term 0.
I20260928 20:19:46.523927 8050 raft_consensus.cc:2454] T 85cae9437e224b4a868c6384caf818b4 P b65a673f17f8487f8e9431585a7cf0d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19c23e487a9a4a5b81937750076082ca in term 0.
I20260928 20:19:46.524101 8117 leader_election.cc:302] T 85cae9437e224b4a868c6384caf818b4 P 19c23e487a9a4a5b81937750076082ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19c23e487a9a4a5b81937750076082ca, 9808c5d773f641e894a7df1d78643909; no voters:
I20260928 20:19:46.524238 8248 raft_consensus.cc:2781] T 85cae9437e224b4a868c6384caf818b4 P 19c23e487a9a4a5b81937750076082ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 20:19:46.524294 8248 raft_consensus.cc:482] T 85cae9437e224b4a868c6384caf818b4 P 19c23e487a9a4a5b81937750076082ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 20:19:46.524315 8248 raft_consensus.cc:3037] T 85cae9437e224b4a868c6384caf818b4 P 19c23e487a9a4a5b81937750076082ca [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:46.525061 8248 raft_consensus.cc:504] T 85cae9437e224b4a868c6384caf818b4 P 19c23e487a9a4a5b81937750076082ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65a673f17f8487f8e9431585a7cf0d0" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 43121 } } peers { permanent_uuid: "19c23e487a9a4a5b81937750076082ca" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 46405 } } peers { permanent_uuid: "9808c5d773f641e894a7df1d78643909" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 33781 } }
I20260928 20:19:46.525204 8248 leader_election.cc:288] T 85cae9437e224b4a868c6384caf818b4 P 19c23e487a9a4a5b81937750076082ca [CANDIDATE]: Term 1 election: Requested vote from peers b65a673f17f8487f8e9431585a7cf0d0 (127.3.18.2:43121), 9808c5d773f641e894a7df1d78643909 (127.3.18.1:33781)
I20260928 20:19:46.525360 7919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85cae9437e224b4a868c6384caf818b4" candidate_uuid: "19c23e487a9a4a5b81937750076082ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9808c5d773f641e894a7df1d78643909"
I20260928 20:19:46.525444 8050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85cae9437e224b4a868c6384caf818b4" candidate_uuid: "19c23e487a9a4a5b81937750076082ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b65a673f17f8487f8e9431585a7cf0d0"
I20260928 20:19:46.525461 7919 raft_consensus.cc:3037] T 85cae9437e224b4a868c6384caf818b4 P 9808c5d773f641e894a7df1d78643909 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:46.525532 8050 raft_consensus.cc:3037] T 85cae9437e224b4a868c6384caf818b4 P b65a673f17f8487f8e9431585a7cf0d0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:46.526242 7919 raft_consensus.cc:2454] T 85cae9437e224b4a868c6384caf818b4 P 9808c5d773f641e894a7df1d78643909 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19c23e487a9a4a5b81937750076082ca in term 1.
I20260928 20:19:46.526317 8050 raft_consensus.cc:2454] T 85cae9437e224b4a868c6384caf818b4 P b65a673f17f8487f8e9431585a7cf0d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19c23e487a9a4a5b81937750076082ca in term 1.
I20260928 20:19:46.526430 8117 leader_election.cc:302] T 85cae9437e224b4a868c6384caf818b4 P 19c23e487a9a4a5b81937750076082ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19c23e487a9a4a5b81937750076082ca, 9808c5d773f641e894a7df1d78643909; no voters:
I20260928 20:19:46.526551 8248 raft_consensus.cc:2781] T 85cae9437e224b4a868c6384caf818b4 P 19c23e487a9a4a5b81937750076082ca [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:19:46.526727 8248 raft_consensus.cc:686] T 85cae9437e224b4a868c6384caf818b4 P 19c23e487a9a4a5b81937750076082ca [term 1 LEADER]: Becoming Leader. State: Replica: 19c23e487a9a4a5b81937750076082ca, State: Running, Role: LEADER
I20260928 20:19:46.526834 8248 consensus_queue.cc:237] T 85cae9437e224b4a868c6384caf818b4 P 19c23e487a9a4a5b81937750076082ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65a673f17f8487f8e9431585a7cf0d0" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 43121 } } peers { permanent_uuid: "19c23e487a9a4a5b81937750076082ca" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 46405 } } peers { permanent_uuid: "9808c5d773f641e894a7df1d78643909" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 33781 } }
I20260928 20:19:46.527733 7776 catalog_manager.cc:5495] T 85cae9437e224b4a868c6384caf818b4 P 19c23e487a9a4a5b81937750076082ca reported cstate change: term changed from 0 to 1, leader changed from <none> to 19c23e487a9a4a5b81937750076082ca (127.3.18.3). New cstate: current_term: 1 leader_uuid: "19c23e487a9a4a5b81937750076082ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b65a673f17f8487f8e9431585a7cf0d0" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 43121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19c23e487a9a4a5b81937750076082ca" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 46405 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9808c5d773f641e894a7df1d78643909" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 33781 } health_report { overall_health: UNKNOWN } } }
I20260928 20:19:46.536435 8240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:46.535425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46624 (local address 127.3.18.3:46405)
0928 20:19:46.535488 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:46.535490 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:46.535499 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:19:46.535537 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:46.535538 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:46.535549 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:46.535614 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:46.536192 (+ 578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:46.536193 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:46.536193 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:19:46.536197 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:46.536212 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:19:46.536216 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:46.536263 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:46.536320 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 20:19:46.536331 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 20:19:46.538286 7919 raft_consensus.cc:1261] T 85cae9437e224b4a868c6384caf818b4 P 9808c5d773f641e894a7df1d78643909 [term 1 FOLLOWER]: Refusing update from remote peer 19c23e487a9a4a5b81937750076082ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:19:46.538306 8050 raft_consensus.cc:1261] T 85cae9437e224b4a868c6384caf818b4 P b65a673f17f8487f8e9431585a7cf0d0 [term 1 FOLLOWER]: Refusing update from remote peer 19c23e487a9a4a5b81937750076082ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:19:46.538677 8251 consensus_queue.cc:1035] T 85cae9437e224b4a868c6384caf818b4 P 19c23e487a9a4a5b81937750076082ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "b65a673f17f8487f8e9431585a7cf0d0" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 43121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:19:46.538820 8248 consensus_queue.cc:1035] T 85cae9437e224b4a868c6384caf818b4 P 19c23e487a9a4a5b81937750076082ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "9808c5d773f641e894a7df1d78643909" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 33781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:19:46.541725 8256 mvcc.cc:205] Tried to move back new op lower bound from 7334407317658324992 to 7334407317614841856. Current Snapshot: MvccSnapshot[applied={T|T < 7334407317658324992}]
I20260928 20:19:46.542323 8255 mvcc.cc:205] Tried to move back new op lower bound from 7334407317658324992 to 7334407317614841856. Current Snapshot: MvccSnapshot[applied={T|T < 7334407317658324992}]
I20260928 20:19:46.543546 7777 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58772:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:19:46.543646 7777 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58772:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:19:46.544428 7777 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c08c01564b1d44bc9be8d30f150daf69: Sending DeleteTablet for 3 replicas of tablet 85cae9437e224b4a868c6384caf818b4
I20260928 20:19:46.544768 7899 tablet_service.cc:1514] Processing DeleteTablet for tablet 85cae9437e224b4a868c6384caf818b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:19:46 UTC) from {username='slave'} at 127.0.0.1:43796
I20260928 20:19:46.544764 8030 tablet_service.cc:1514] Processing DeleteTablet for tablet 85cae9437e224b4a868c6384caf818b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:19:46 UTC) from {username='slave'} at 127.0.0.1:41490
I20260928 20:19:46.544780 8161 tablet_service.cc:1514] Processing DeleteTablet for tablet 85cae9437e224b4a868c6384caf818b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:19:46 UTC) from {username='slave'} at 127.0.0.1:46612
I20260928 20:19:46.545357 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 7837
I20260928 20:19:46.552013 8266 tablet_replica.cc:330] T 85cae9437e224b4a868c6384caf818b4 P b65a673f17f8487f8e9431585a7cf0d0: stopping tablet replica
I20260928 20:19:46.552196 8266 raft_consensus.cc:2229] T 85cae9437e224b4a868c6384caf818b4 P b65a673f17f8487f8e9431585a7cf0d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:19:46.552277 8264 tablet_replica.cc:330] T 85cae9437e224b4a868c6384caf818b4 P 19c23e487a9a4a5b81937750076082ca: stopping tablet replica
I20260928 20:19:46.552357 8266 raft_consensus.cc:2258] T 85cae9437e224b4a868c6384caf818b4 P b65a673f17f8487f8e9431585a7cf0d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:19:46.552440 8264 raft_consensus.cc:2229] T 85cae9437e224b4a868c6384caf818b4 P 19c23e487a9a4a5b81937750076082ca [term 1 LEADER]: Raft consensus shutting down.
I20260928 20:19:46.552592 8264 raft_consensus.cc:2258] T 85cae9437e224b4a868c6384caf818b4 P 19c23e487a9a4a5b81937750076082ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:19:46.552911 8266 ts_tablet_manager.cc:1905] T 85cae9437e224b4a868c6384caf818b4 P b65a673f17f8487f8e9431585a7cf0d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:19:46.553108 8264 ts_tablet_manager.cc:1905] T 85cae9437e224b4a868c6384caf818b4 P 19c23e487a9a4a5b81937750076082ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:19:46.554842 8266 ts_tablet_manager.cc:1918] T 85cae9437e224b4a868c6384caf818b4 P b65a673f17f8487f8e9431585a7cf0d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:19:46.554919 8266 log.cc:1196] T 85cae9437e224b4a868c6384caf818b4 P b65a673f17f8487f8e9431585a7cf0d0: Deleting WAL directory at /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790626764250411-3144-0/minicluster-data/ts-1/wal/wals/85cae9437e224b4a868c6384caf818b4
I20260928 20:19:46.555179 8266 ts_tablet_manager.cc:1939] T 85cae9437e224b4a868c6384caf818b4 P b65a673f17f8487f8e9431585a7cf0d0: Deleting consensus metadata
I20260928 20:19:46.555454 8264 ts_tablet_manager.cc:1918] T 85cae9437e224b4a868c6384caf818b4 P 19c23e487a9a4a5b81937750076082ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:19:46.555537 8264 log.cc:1196] T 85cae9437e224b4a868c6384caf818b4 P 19c23e487a9a4a5b81937750076082ca: Deleting WAL directory at /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790626764250411-3144-0/minicluster-data/ts-2/wal/wals/85cae9437e224b4a868c6384caf818b4
I20260928 20:19:46.555668 7761 catalog_manager.cc:4841] TS b65a673f17f8487f8e9431585a7cf0d0 (127.3.18.2:43121): tablet 85cae9437e224b4a868c6384caf818b4 (table test-table [id=f5aaf9a01ae4400cba9b40e0c2b9e281]) successfully deleted
I20260928 20:19:46.555886 8264 ts_tablet_manager.cc:1939] T 85cae9437e224b4a868c6384caf818b4 P 19c23e487a9a4a5b81937750076082ca: Deleting consensus metadata
I20260928 20:19:46.556336 7761 catalog_manager.cc:4841] TS 19c23e487a9a4a5b81937750076082ca (127.3.18.3:46405): tablet 85cae9437e224b4a868c6384caf818b4 (table test-table [id=f5aaf9a01ae4400cba9b40e0c2b9e281]) successfully deleted
W20260928 20:19:46.556442 7764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.18.1:33781 (error 108)
I20260928 20:19:46.556711 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 7968
I20260928 20:19:46.558159 8239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:46.557991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.1:33781 (local address 127.0.0.1:43808)
0928 20:19:46.558068 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:46.558117 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.1:33781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 20:19:46.558293 7764 catalog_manager.cc:4584] TS 9808c5d773f641e894a7df1d78643909 (127.3.18.1:33781): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 85cae9437e224b4a868c6384caf818b4: Network error: Client connection negotiation failed: client connection to 127.3.18.1:33781: connect: Connection refused (error 111)
I20260928 20:19:46.561549 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 8099
I20260928 20:19:46.566462 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 7746
2026-09-28T20:19:46Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (624 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 20:19:46.586817 3144 mini_oidc.cc:152] Starting JWKS server
I20260928 20:19:46.588080 3144 webserver.cc:460] Webserver started at https://127.0.0.1:33369/ using document root <none> and password file <none>
I20260928 20:19:46.588168 3144 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 20:19:46.588389 3144 webserver.cc:460] Webserver started at http://127.0.0.1:40879/ using document root <none> and password file <none>
2026-09-28T20:19:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:19:46Z Disabled control of system clock
I20260928 20:19:46.596900 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790626764250411-3144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790626764250411-3144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790626764250411-3144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:46853
--webserver_interface=127.3.18.62
--webserver_port=0
--builtin_ntp_servers=127.3.18.20:44365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.18.62:46853
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33369/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790626764250411-3144-0/testchainca.cert with env {}
W20260928 20:19:46.678763 8273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:19:46.678995 8273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:19:46.679045 8273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:19:46.680378 8273 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:19:46.680434 8273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:19:46.680447 8273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:19:46.680460 8273 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:19:46.680472 8273 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 20:19:46.680485 8273 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 20:19:46.680495 8273 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33369/jwks.json
I20260928 20:19:46.682281 8273 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:44365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790626764250411-3144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790626764250411-3144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.18.62:46853
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:46853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33369/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.18.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790626764250411-3144-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790626764250411-3144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:19:46.682606 8273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:19:46.682853 8273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:19:46.687999 8279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:46.688024 8278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:46.688114 8273 server_base.cc:1034] running on GCE node
W20260928 20:19:46.687999 8281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:46.688535 8273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:19:46.688851 8273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:19:46.690065 8273 hybrid_clock.cc:648] HybridClock initialized: now 1790626786690041 us; error 38 us; skew 500 ppm
I20260928 20:19:46.691530 8273 webserver.cc:460] Webserver started at http://127.3.18.62:40351/ using document root <none> and password file <none>
I20260928 20:19:46.691982 8273 fs_manager.cc:359] Metadata directory not provided
I20260928 20:19:46.692041 8273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:19:46.692201 8273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:19:46.693440 8273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790626764250411-3144-0/minicluster-data/master-0/data/instance:
uuid: "d39755ffc6454c04a637275fdd4ff019"
format_stamp: "Formatted at 2026-09-28 20:19:46 on dist-test-slave-b83v"
server_key: "c8da1ad8168502fbbdbbcc8ba02544de"
server_key_iv: "ac87884b7b58cfc3b252041d08c7436f"
server_key_version: "encryptionkey@0"
I20260928 20:19:46.693868 8273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790626764250411-3144-0/minicluster-data/master-0/wal/instance:
uuid: "d39755ffc6454c04a637275fdd4ff019"
format_stamp: "Formatted at 2026-09-28 20:19:46 on dist-test-slave-b83v"
server_key: "c8da1ad8168502fbbdbbcc8ba02544de"
server_key_iv: "ac87884b7b58cfc3b252041d08c7436f"
server_key_version: "encryptionkey@0"
I20260928 20:19:46.695290 8273 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 20:19:46.696185 8287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:46.696432 8273 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 20:19:46.696511 8273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790626764250411-3144-0/minicluster-data/master-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790626764250411-3144-0/minicluster-data/master-0/wal
uuid: "d39755ffc6454c04a637275fdd4ff019"
format_stamp: "Formatted at 2026-09-28 20:19:46 on dist-test-slave-b83v"
server_key: "c8da1ad8168502fbbdbbcc8ba02544de"
server_key_iv: "ac87884b7b58cfc3b252041d08c7436f"
server_key_version: "encryptionkey@0"
I20260928 20:19:46.696609 8273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790626764250411-3144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790626764250411-3144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790626764250411-3144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:19:46.708278 8273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:19:46.708631 8273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:19:46.709003 8273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:19:46.713438 8273 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.62:46853
I20260928 20:19:46.713480 8339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.62:46853 every 8 connection(s)
I20260928 20:19:46.713874 8273 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
I20260928 20:19:46.714993 8340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:46.717583 8340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d39755ffc6454c04a637275fdd4ff019: Bootstrap starting.
I20260928 20:19:46.718255 8340 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d39755ffc6454c04a637275fdd4ff019: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:46.718591 8340 log.cc:824] T 00000000000000000000000000000000 P d39755ffc6454c04a637275fdd4ff019: Log is configured to *not* fsync() on all Append() calls
I20260928 20:19:46.719427 8340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d39755ffc6454c04a637275fdd4ff019: No bootstrap required, opened a new log
I20260928 20:19:46.722078 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 8273
I20260928 20:19:46.722198 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790626764250411-3144-0/minicluster-data/master-0/wal/instance
I20260928 20:19:46.722450 3144 external_mini_cluster.cc:1428] Setting key e2f030f23caf28d19791e6a18a0f6ef4
I20260928 20:19:46.722334 8340 raft_consensus.cc:348] T 00000000000000000000000000000000 P d39755ffc6454c04a637275fdd4ff019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d39755ffc6454c04a637275fdd4ff019" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 46853 } }
I20260928 20:19:46.722491 8340 raft_consensus.cc:374] T 00000000000000000000000000000000 P d39755ffc6454c04a637275fdd4ff019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:46.722515 8340 raft_consensus.cc:729] T 00000000000000000000000000000000 P d39755ffc6454c04a637275fdd4ff019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d39755ffc6454c04a637275fdd4ff019, State: Initialized, Role: FOLLOWER
I20260928 20:19:46.722702 8340 consensus_queue.cc:260] T 00000000000000000000000000000000 P d39755ffc6454c04a637275fdd4ff019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d39755ffc6454c04a637275fdd4ff019" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 46853 } }
I20260928 20:19:46.723212 8340 raft_consensus.cc:388] T 00000000000000000000000000000000 P d39755ffc6454c04a637275fdd4ff019 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:19:46.723278 8340 raft_consensus.cc:482] T 00000000000000000000000000000000 P d39755ffc6454c04a637275fdd4ff019 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:19:46.723325 8340 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d39755ffc6454c04a637275fdd4ff019 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:46.724263 8340 raft_consensus.cc:504] T 00000000000000000000000000000000 P d39755ffc6454c04a637275fdd4ff019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d39755ffc6454c04a637275fdd4ff019" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 46853 } }
I20260928 20:19:46.724398 8340 leader_election.cc:302] T 00000000000000000000000000000000 P d39755ffc6454c04a637275fdd4ff019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d39755ffc6454c04a637275fdd4ff019; no voters:
I20260928 20:19:46.724614 8340 leader_election.cc:288] T 00000000000000000000000000000000 P d39755ffc6454c04a637275fdd4ff019 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:19:46.724678 8345 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d39755ffc6454c04a637275fdd4ff019 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:19:46.724833 8345 raft_consensus.cc:686] T 00000000000000000000000000000000 P d39755ffc6454c04a637275fdd4ff019 [term 1 LEADER]: Becoming Leader. State: Replica: d39755ffc6454c04a637275fdd4ff019, State: Running, Role: LEADER
I20260928 20:19:46.724923 8345 consensus_queue.cc:237] T 00000000000000000000000000000000 P d39755ffc6454c04a637275fdd4ff019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d39755ffc6454c04a637275fdd4ff019" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 46853 } }
I20260928 20:19:46.724961 8340 sys_catalog.cc:564] T 00000000000000000000000000000000 P d39755ffc6454c04a637275fdd4ff019 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:19:46.725530 8346 sys_catalog.cc:455] T 00000000000000000000000000000000 P d39755ffc6454c04a637275fdd4ff019 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d39755ffc6454c04a637275fdd4ff019" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d39755ffc6454c04a637275fdd4ff019" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 46853 } } }
I20260928 20:19:46.725518 8345 sys_catalog.cc:455] T 00000000000000000000000000000000 P d39755ffc6454c04a637275fdd4ff019 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d39755ffc6454c04a637275fdd4ff019. Latest consensus state: current_term: 1 leader_uuid: "d39755ffc6454c04a637275fdd4ff019" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d39755ffc6454c04a637275fdd4ff019" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 46853 } } }
I20260928 20:19:46.725710 8346 sys_catalog.cc:458] T 00000000000000000000000000000000 P d39755ffc6454c04a637275fdd4ff019 [sys.catalog]: This master's current role is: LEADER
I20260928 20:19:46.725733 8345 sys_catalog.cc:458] T 00000000000000000000000000000000 P d39755ffc6454c04a637275fdd4ff019 [sys.catalog]: This master's current role is: LEADER
I20260928 20:19:46.726065 8354 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:19:46.726676 8354 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:19:46.728498 8354 catalog_manager.cc:1347] Generated new cluster ID: 605466f534e247b2bbaedfd338b1c62e
I20260928 20:19:46.728555 8354 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:19:46.730775 8344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:46.723011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40718 (local address 127.3.18.62:46853)
0928 20:19:46.723256 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:46.723267 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:46.723304 (+ 37us) server_negotiation.cc:407] Connection header received
0928 20:19:46.723444 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:46.723452 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:46.723535 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:46.723728 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:46.724106 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:46.726357 (+ 2251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:46.727991 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:46.728720 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:46.728907 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:46.729747 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:46.729751 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:46.729752 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:19:46.730008 (+ 256us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:46.730057 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:19:46.730074 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:46.730374 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:46.730415 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 20:19:46.730525 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":121,"threads_started":1}
I20260928 20:19:46.734760 3144 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 20:19:46.760166 8354 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:19:46.760783 8354 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:19:46.770133 8354 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d39755ffc6454c04a637275fdd4ff019: Generated new TSK 0
I20260928 20:19:46.770371 8354 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 20:19:46.796067 8370 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 20:19:46.796295 8370 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 20:19:46.796368 8344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:46.778903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40730 (local address 127.3.18.62:46853)
0928 20:19:46.778975 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:46.778977 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:46.779111 (+ 134us) server_negotiation.cc:407] Connection header received
0928 20:19:46.779194 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:46.779195 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:46.779209 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:46.779299 (+ 90us) server_negotiation.cc:226] Negotiated authn=JWT
0928 20:19:46.779559 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:46.780357 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:46.781409 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:46.781739 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:46.781811 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:46.782145 (+ 334us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 20:19:46.796122 (+ 13977us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33369/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 20:19:46.796292 (+ 170us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40730: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33369/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":44}
W20260928 20:19:46.796432 8344 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40730: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33369/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260928 20:19:46.797421 8369 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:19:46.778768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:46853 (local address 127.0.0.1:40730)
0928 20:19:46.779037 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:46.779048 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:46.779112 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:46.779402 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:46.779403 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:46.779406 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0928 20:19:46.779473 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:46.779475 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:46.780495 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:46.780496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:46.781315 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:46.781319 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:46.781932 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:46.781937 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:46.781993 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:46.782019 (+ 26us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 20:19:46.797005 (+ 14986us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33369/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 20:19:46.797139 (+ 134us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.18.62:46853: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33369/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":169,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.3.18.62:46853: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33369/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 20:19:46.798039 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 8273
2026-09-28T20:19:46Z chronyd exiting
I20260928 20:19:46.835155 3144 test_util.cc:175] -----------------------------------------------
I20260928 20:19:46.835246 3144 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790626764250411-3144-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (249 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 20:19:46.836408 3144 mini_oidc.cc:152] Starting JWKS server
I20260928 20:19:46.837128 3144 webserver.cc:460] Webserver started at https://127.0.0.1:39371/ using document root <none> and password file <none>
I20260928 20:19:46.837177 3144 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 20:19:46.837345 3144 webserver.cc:460] Webserver started at http://127.0.0.1:36525/ using document root <none> and password file <none>
2026-09-28T20:19:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:19:46Z Disabled control of system clock
I20260928 20:19:46.845976 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790626764250411-3144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790626764250411-3144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790626764250411-3144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:41465
--webserver_interface=127.3.18.62
--webserver_port=0
--builtin_ntp_servers=127.3.18.20:36093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.18.62:41465
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39371/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790626764250411-3144-0/testchainca.cert with env {}
W20260928 20:19:46.932492 8376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:19:46.932780 8376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:19:46.932825 8376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:19:46.934226 8376 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:19:46.934311 8376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:19:46.934334 8376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:19:46.934360 8376 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:19:46.934392 8376 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 20:19:46.934419 8376 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 20:19:46.934448 8376 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39371/jwks.json
I20260928 20:19:46.936411 8376 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:36093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790626764250411-3144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790626764250411-3144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.18.62:41465
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:41465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39371/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.18.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790626764250411-3144-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790626764250411-3144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:19:46.936738 8376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:19:46.937001 8376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:19:46.942615 8382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:46.942744 8384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:46.942910 8376 server_base.cc:1034] running on GCE node
W20260928 20:19:46.942991 8381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:46.943287 8376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:19:46.943629 8376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:19:46.944840 8376 hybrid_clock.cc:648] HybridClock initialized: now 1790626786944799 us; error 83 us; skew 500 ppm
I20260928 20:19:46.946321 8376 webserver.cc:460] Webserver started at http://127.3.18.62:44717/ using document root <none> and password file <none>
I20260928 20:19:46.946657 8376 fs_manager.cc:359] Metadata directory not provided
I20260928 20:19:46.946739 8376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:19:46.946944 8376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:19:46.948163 8376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790626764250411-3144-0/minicluster-data/master-0/data/instance:
uuid: "1ba790e6ea9d49589bf39b1543b7acb5"
format_stamp: "Formatted at 2026-09-28 20:19:46 on dist-test-slave-b83v"
server_key: "0b5f04a049c8be38fdb611298911396c"
server_key_iv: "2805afe204dc3c9bb50bdb2de2464828"
server_key_version: "encryptionkey@0"
I20260928 20:19:46.948536 8376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790626764250411-3144-0/minicluster-data/master-0/wal/instance:
uuid: "1ba790e6ea9d49589bf39b1543b7acb5"
format_stamp: "Formatted at 2026-09-28 20:19:46 on dist-test-slave-b83v"
server_key: "0b5f04a049c8be38fdb611298911396c"
server_key_iv: "2805afe204dc3c9bb50bdb2de2464828"
server_key_version: "encryptionkey@0"
I20260928 20:19:46.949977 8376 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 20:19:46.950861 8390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:46.951049 8376 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:19:46.951161 8376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790626764250411-3144-0/minicluster-data/master-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790626764250411-3144-0/minicluster-data/master-0/wal
uuid: "1ba790e6ea9d49589bf39b1543b7acb5"
format_stamp: "Formatted at 2026-09-28 20:19:46 on dist-test-slave-b83v"
server_key: "0b5f04a049c8be38fdb611298911396c"
server_key_iv: "2805afe204dc3c9bb50bdb2de2464828"
server_key_version: "encryptionkey@0"
I20260928 20:19:46.951242 8376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790626764250411-3144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790626764250411-3144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790626764250411-3144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:19:46.975047 8376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:19:46.975430 8376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:19:46.975594 8376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:19:46.979892 8376 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.62:41465
I20260928 20:19:46.979940 8442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.62:41465 every 8 connection(s)
I20260928 20:19:46.980360 8376 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
I20260928 20:19:46.981514 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 8376
I20260928 20:19:46.981592 8443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:46.981623 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790626764250411-3144-0/minicluster-data/master-0/wal/instance
I20260928 20:19:46.981818 3144 external_mini_cluster.cc:1428] Setting key 21752e8a63e29412d79c3b03a33b1346
I20260928 20:19:46.985129 8443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ba790e6ea9d49589bf39b1543b7acb5: Bootstrap starting.
I20260928 20:19:46.985908 8443 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ba790e6ea9d49589bf39b1543b7acb5: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:46.986234 8443 log.cc:824] T 00000000000000000000000000000000 P 1ba790e6ea9d49589bf39b1543b7acb5: Log is configured to *not* fsync() on all Append() calls
I20260928 20:19:46.987062 8443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ba790e6ea9d49589bf39b1543b7acb5: No bootstrap required, opened a new log
I20260928 20:19:46.988571 8446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:46.982423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40640 (local address 127.3.18.62:41465)
0928 20:19:46.982673 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:46.982682 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:46.982718 (+ 36us) server_negotiation.cc:407] Connection header received
0928 20:19:46.983367 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:46.983376 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:46.983457 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:46.983595 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:46.983877 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:46.984987 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:46.986233 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:46.986713 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:46.986811 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:46.987606 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:46.987610 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:46.987610 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:19:46.987892 (+ 282us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:46.987936 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:19:46.987953 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:46.988027 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:46.988139 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 20:19:46.988313 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":115,"threads_started":1}
I20260928 20:19:46.989606 8443 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ba790e6ea9d49589bf39b1543b7acb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba790e6ea9d49589bf39b1543b7acb5" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 41465 } }
I20260928 20:19:46.989771 3144 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 20:19:46.989811 8443 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ba790e6ea9d49589bf39b1543b7acb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:46.989863 8443 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ba790e6ea9d49589bf39b1543b7acb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ba790e6ea9d49589bf39b1543b7acb5, State: Initialized, Role: FOLLOWER
I20260928 20:19:46.989979 8443 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ba790e6ea9d49589bf39b1543b7acb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba790e6ea9d49589bf39b1543b7acb5" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 41465 } }
I20260928 20:19:46.990051 8443 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ba790e6ea9d49589bf39b1543b7acb5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:19:46.990078 8443 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ba790e6ea9d49589bf39b1543b7acb5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:19:46.990121 8443 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ba790e6ea9d49589bf39b1543b7acb5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:46.990947 8443 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ba790e6ea9d49589bf39b1543b7acb5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba790e6ea9d49589bf39b1543b7acb5" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 41465 } }
I20260928 20:19:46.991046 8443 leader_election.cc:302] T 00000000000000000000000000000000 P 1ba790e6ea9d49589bf39b1543b7acb5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ba790e6ea9d49589bf39b1543b7acb5; no voters:
I20260928 20:19:46.991240 8443 leader_election.cc:288] T 00000000000000000000000000000000 P 1ba790e6ea9d49589bf39b1543b7acb5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:19:46.991317 8448 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ba790e6ea9d49589bf39b1543b7acb5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:19:46.991477 8448 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ba790e6ea9d49589bf39b1543b7acb5 [term 1 LEADER]: Becoming Leader. State: Replica: 1ba790e6ea9d49589bf39b1543b7acb5, State: Running, Role: LEADER
I20260928 20:19:46.991499 8443 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ba790e6ea9d49589bf39b1543b7acb5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:19:46.991576 8448 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ba790e6ea9d49589bf39b1543b7acb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba790e6ea9d49589bf39b1543b7acb5" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 41465 } }
I20260928 20:19:46.992034 8449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ba790e6ea9d49589bf39b1543b7acb5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ba790e6ea9d49589bf39b1543b7acb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba790e6ea9d49589bf39b1543b7acb5" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 41465 } } }
I20260928 20:19:46.992147 8449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ba790e6ea9d49589bf39b1543b7acb5 [sys.catalog]: This master's current role is: LEADER
I20260928 20:19:46.992115 8450 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ba790e6ea9d49589bf39b1543b7acb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ba790e6ea9d49589bf39b1543b7acb5. Latest consensus state: current_term: 1 leader_uuid: "1ba790e6ea9d49589bf39b1543b7acb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba790e6ea9d49589bf39b1543b7acb5" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 41465 } } }
I20260928 20:19:46.992297 8450 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ba790e6ea9d49589bf39b1543b7acb5 [sys.catalog]: This master's current role is: LEADER
I20260928 20:19:46.992466 8457 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:19:46.993002 8457 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:19:46.994890 8457 catalog_manager.cc:1347] Generated new cluster ID: 95f9b92cdcc44e8f893f9da782a78e7e
I20260928 20:19:46.994943 8457 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:19:47.003014 8457 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:19:47.003516 8457 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:19:47.015872 8457 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ba790e6ea9d49589bf39b1543b7acb5: Generated new TSK 0
I20260928 20:19:47.016125 8457 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 20:19:47.044648 8473 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 20:19:47.044775 8473 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 20:19:47.045249 8446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:47.028227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40644 (local address 127.3.18.62:41465)
0928 20:19:47.028297 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:47.028300 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:47.028435 (+ 135us) server_negotiation.cc:407] Connection header received
0928 20:19:47.028512 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:47.028512 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:47.028523 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:47.028584 (+ 61us) server_negotiation.cc:226] Negotiated authn=JWT
0928 20:19:47.028748 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:47.029244 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:47.030455 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:47.030658 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:47.030735 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:47.030979 (+ 244us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 20:19:47.045004 (+ 14025us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39371/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 20:19:47.045182 (+ 178us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40644: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39371/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":43}
W20260928 20:19:47.045429 8472 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:19:47.028171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:41465 (local address 127.0.0.1:40644)
0928 20:19:47.028334 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:47.028346 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:47.028434 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:47.028607 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:47.028608 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:47.028609 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0928 20:19:47.028671 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:47.028673 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:47.029319 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:47.029320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:47.030300 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:47.030305 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:47.030753 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:47.030758 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:47.030827 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:47.030833 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 20:19:47.045167 (+ 14334us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39371/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 20:19:47.045319 (+ 152us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.18.62:41465: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39371/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":106,"threads_started":1}
W20260928 20:19:47.045341 8446 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40644: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39371/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260928 20:19:47.045780 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 8376
2026-09-28T20:19:47Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (247 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 20:19:47.084055 3144 mini_oidc.cc:152] Starting JWKS server
I20260928 20:19:47.084753 3144 webserver.cc:460] Webserver started at https://127.0.0.1:44307/ using document root <none> and password file <none>
I20260928 20:19:47.084830 3144 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 20:19:47.084995 3144 webserver.cc:460] Webserver started at http://127.0.0.1:43603/ using document root <none> and password file <none>
2026-09-28T20:19:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:19:47Z Disabled control of system clock
I20260928 20:19:47.093668 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790626764250411-3144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790626764250411-3144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790626764250411-3144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:43165
--webserver_interface=127.3.18.62
--webserver_port=0
--builtin_ntp_servers=127.3.18.20:46825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.18.62:43165
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44307/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 20:19:47.175993 8479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:19:47.176251 8479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:19:47.176275 8479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:19:47.177482 8479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:19:47.177526 8479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:19:47.177539 8479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:19:47.177551 8479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:19:47.177563 8479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 20:19:47.177575 8479 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 20:19:47.177587 8479 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44307/jwks.json
I20260928 20:19:47.179152 8479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:46825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790626764250411-3144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790626764250411-3144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.18.62:43165
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:43165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44307/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.18.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790626764250411-3144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:19:47.179386 8479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:19:47.179678 8479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:19:47.185230 8484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:47.185230 8487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:47.185222 8485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:47.185585 8479 server_base.cc:1034] running on GCE node
I20260928 20:19:47.185847 8479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:19:47.186183 8479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:19:47.187469 8479 hybrid_clock.cc:648] HybridClock initialized: now 1790626787187423 us; error 61 us; skew 500 ppm
I20260928 20:19:47.189100 8479 webserver.cc:460] Webserver started at http://127.3.18.62:44925/ using document root <none> and password file <none>
I20260928 20:19:47.189426 8479 fs_manager.cc:359] Metadata directory not provided
I20260928 20:19:47.189508 8479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:19:47.189683 8479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:19:47.191069 8479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790626764250411-3144-0/minicluster-data/master-0/data/instance:
uuid: "9b0e6985db4141d18cd59c68e47962c3"
format_stamp: "Formatted at 2026-09-28 20:19:47 on dist-test-slave-b83v"
server_key: "d4598b9626e9857b3c11c0144ed392d1"
server_key_iv: "59f4fba1c8fd131ff4c19594b8aac398"
server_key_version: "encryptionkey@0"
I20260928 20:19:47.191534 8479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790626764250411-3144-0/minicluster-data/master-0/wal/instance:
uuid: "9b0e6985db4141d18cd59c68e47962c3"
format_stamp: "Formatted at 2026-09-28 20:19:47 on dist-test-slave-b83v"
server_key: "d4598b9626e9857b3c11c0144ed392d1"
server_key_iv: "59f4fba1c8fd131ff4c19594b8aac398"
server_key_version: "encryptionkey@0"
I20260928 20:19:47.193189 8479 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:19:47.194239 8493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:47.194729 8479 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:19:47.194943 8479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790626764250411-3144-0/minicluster-data/master-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790626764250411-3144-0/minicluster-data/master-0/wal
uuid: "9b0e6985db4141d18cd59c68e47962c3"
format_stamp: "Formatted at 2026-09-28 20:19:47 on dist-test-slave-b83v"
server_key: "d4598b9626e9857b3c11c0144ed392d1"
server_key_iv: "59f4fba1c8fd131ff4c19594b8aac398"
server_key_version: "encryptionkey@0"
I20260928 20:19:47.195045 8479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790626764250411-3144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790626764250411-3144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790626764250411-3144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:19:47.206732 8479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:19:47.207155 8479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:19:47.207333 8479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:19:47.211987 8479 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.62:43165
I20260928 20:19:47.212039 8545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.62:43165 every 8 connection(s)
I20260928 20:19:47.212414 8479 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
I20260928 20:19:47.213101 8546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:47.215919 8546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b0e6985db4141d18cd59c68e47962c3: Bootstrap starting.
I20260928 20:19:47.216478 8546 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b0e6985db4141d18cd59c68e47962c3: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:47.216748 8546 log.cc:824] T 00000000000000000000000000000000 P 9b0e6985db4141d18cd59c68e47962c3: Log is configured to *not* fsync() on all Append() calls
I20260928 20:19:47.217521 8546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b0e6985db4141d18cd59c68e47962c3: No bootstrap required, opened a new log
I20260928 20:19:47.218537 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 8479
I20260928 20:19:47.218621 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790626764250411-3144-0/minicluster-data/master-0/wal/instance
I20260928 20:19:47.218837 3144 external_mini_cluster.cc:1428] Setting key fe73a1bc0cc3af51163bea3e64f9b8fb
I20260928 20:19:47.219700 8546 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b0e6985db4141d18cd59c68e47962c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b0e6985db4141d18cd59c68e47962c3" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 43165 } }
I20260928 20:19:47.219983 8546 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b0e6985db4141d18cd59c68e47962c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:47.220036 8546 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b0e6985db4141d18cd59c68e47962c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b0e6985db4141d18cd59c68e47962c3, State: Initialized, Role: FOLLOWER
I20260928 20:19:47.220227 8546 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b0e6985db4141d18cd59c68e47962c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b0e6985db4141d18cd59c68e47962c3" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 43165 } }
I20260928 20:19:47.220333 8546 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b0e6985db4141d18cd59c68e47962c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:19:47.220386 8546 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b0e6985db4141d18cd59c68e47962c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:19:47.220438 8546 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b0e6985db4141d18cd59c68e47962c3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:47.221174 8546 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b0e6985db4141d18cd59c68e47962c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b0e6985db4141d18cd59c68e47962c3" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 43165 } }
I20260928 20:19:47.221314 8546 leader_election.cc:302] T 00000000000000000000000000000000 P 9b0e6985db4141d18cd59c68e47962c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b0e6985db4141d18cd59c68e47962c3; no voters:
I20260928 20:19:47.221542 8546 leader_election.cc:288] T 00000000000000000000000000000000 P 9b0e6985db4141d18cd59c68e47962c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:19:47.221755 8552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b0e6985db4141d18cd59c68e47962c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b0e6985db4141d18cd59c68e47962c3" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 43165 } } }
I20260928 20:19:47.221781 8551 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b0e6985db4141d18cd59c68e47962c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:19:47.221921 8552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b0e6985db4141d18cd59c68e47962c3 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 20:19:47.222090 8551 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b0e6985db4141d18cd59c68e47962c3 [term 1 LEADER]: Becoming Leader. State: Replica: 9b0e6985db4141d18cd59c68e47962c3, State: Running, Role: LEADER
I20260928 20:19:47.221849 8546 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b0e6985db4141d18cd59c68e47962c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:19:47.222204 8551 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b0e6985db4141d18cd59c68e47962c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b0e6985db4141d18cd59c68e47962c3" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 43165 } }
I20260928 20:19:47.222622 8552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b0e6985db4141d18cd59c68e47962c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b0e6985db4141d18cd59c68e47962c3. Latest consensus state: current_term: 1 leader_uuid: "9b0e6985db4141d18cd59c68e47962c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b0e6985db4141d18cd59c68e47962c3" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 43165 } } }
I20260928 20:19:47.222719 8552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b0e6985db4141d18cd59c68e47962c3 [sys.catalog]: This master's current role is: LEADER
I20260928 20:19:47.223309 8556 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:19:47.223887 8556 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:19:47.226354 8550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:47.219898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58522 (local address 127.3.18.62:43165)
0928 20:19:47.220146 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:47.220156 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:47.220201 (+ 45us) server_negotiation.cc:407] Connection header received
0928 20:19:47.220306 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:47.220312 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:47.220404 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:47.220567 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:47.220791 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:47.222858 (+ 2067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:47.224102 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:47.224574 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:47.224736 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:47.225494 (+ 758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:47.225499 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:47.225499 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:19:47.225721 (+ 222us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:47.225766 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:19:47.225782 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:47.225874 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:47.225962 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 20:19:47.226056 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":107,"threads_started":1}
I20260928 20:19:47.227262 8556 catalog_manager.cc:1347] Generated new cluster ID: 92bda692b30e4ee8a5f8bb6e5561928d
I20260928 20:19:47.227321 8556 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:19:47.227455 3144 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 20:19:47.237397 8556 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:19:47.238170 8556 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:19:47.242954 8556 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b0e6985db4141d18cd59c68e47962c3: Generated new TSK 0
I20260928 20:19:47.243177 8556 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 20:19:47.261264 8550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:47.257405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58526 (local address 127.3.18.62:43165)
0928 20:19:47.257450 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:47.257455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:47.257517 (+ 62us) server_negotiation.cc:407] Connection header received
0928 20:19:47.257609 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:47.257610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:47.257620 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:47.257680 (+ 60us) server_negotiation.cc:226] Negotiated authn=JWT
0928 20:19:47.257839 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:47.258394 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:47.259569 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:47.259755 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:47.259872 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:47.260094 (+ 222us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 20:19:47.261025 (+ 931us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44307/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 20:19:47.261183 (+ 158us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58526: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44307/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 20:19:47.261323 8575 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:19:47.257274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:43165 (local address 127.0.0.1:58526)
0928 20:19:47.257454 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:47.257466 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:47.257537 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:47.257703 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:47.257704 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:47.257706 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 20:19:47.257770 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:47.257772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:47.258489 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:47.258490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:47.259475 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:47.259477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:47.259888 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:47.259896 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:47.259960 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:47.259966 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 20:19:47.261132 (+ 1166us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44307/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 20:19:47.261229 (+ 97us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.18.62:43165: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44307/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":100,"threads_started":1}
W20260928 20:19:47.261408 8550 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58526: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44307/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260928 20:19:47.261852 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 8479
E20260928 20:19:47.273738 8576 webserver.cc:533] Webserver: sslize failed:
E20260928 20:19:47.273823 8576 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-28T20:19:47Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (216 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T20:19:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:19:47Z Disabled control of system clock
I20260928 20:19:47.308349 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790626764250411-3144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790626764250411-3144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790626764250411-3144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:39211
--webserver_interface=127.3.18.62
--webserver_port=0
--builtin_ntp_servers=127.3.18.20:36175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.18.62:39211
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790626764250411-3144-0/testchainca.cert with env {}
W20260928 20:19:47.401695 8580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:19:47.401939 8580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:19:47.401966 8580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:19:47.403431 8580 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:19:47.403484 8580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:19:47.403499 8580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:19:47.403513 8580 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:19:47.403527 8580 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 20:19:47.403541 8580 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 20:19:47.403553 8580 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 20:19:47.405366 8580 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:36175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790626764250411-3144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790626764250411-3144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.18.62:39211
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:39211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.18.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790626764250411-3144-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790626764250411-3144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:19:47.405665 8580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:19:47.405932 8580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:19:47.411669 8585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:47.411688 8580 server_base.cc:1034] running on GCE node
W20260928 20:19:47.411696 8586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:47.411669 8588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:47.412117 8580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:19:47.412482 8580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:19:47.413672 8580 hybrid_clock.cc:648] HybridClock initialized: now 1790626787413665 us; error 37 us; skew 500 ppm
I20260928 20:19:47.414945 8580 webserver.cc:460] Webserver started at http://127.3.18.62:42755/ using document root <none> and password file <none>
I20260928 20:19:47.415205 8580 fs_manager.cc:359] Metadata directory not provided
I20260928 20:19:47.415271 8580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:19:47.415455 8580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:19:47.416623 8580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790626764250411-3144-0/minicluster-data/master-0/data/instance:
uuid: "a2ce021e04cb407590760f7777224f8b"
format_stamp: "Formatted at 2026-09-28 20:19:47 on dist-test-slave-b83v"
server_key: "70451267f0b570f2e6b820ffd60aafe2"
server_key_iv: "9786e4b80b7f5e1bbaee6aadeb838814"
server_key_version: "encryptionkey@0"
I20260928 20:19:47.416972 8580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790626764250411-3144-0/minicluster-data/master-0/wal/instance:
uuid: "a2ce021e04cb407590760f7777224f8b"
format_stamp: "Formatted at 2026-09-28 20:19:47 on dist-test-slave-b83v"
server_key: "70451267f0b570f2e6b820ffd60aafe2"
server_key_iv: "9786e4b80b7f5e1bbaee6aadeb838814"
server_key_version: "encryptionkey@0"
I20260928 20:19:47.418251 8580 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 20:19:47.418978 8594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:47.419162 8580 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 20:19:47.419267 8580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790626764250411-3144-0/minicluster-data/master-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790626764250411-3144-0/minicluster-data/master-0/wal
uuid: "a2ce021e04cb407590760f7777224f8b"
format_stamp: "Formatted at 2026-09-28 20:19:47 on dist-test-slave-b83v"
server_key: "70451267f0b570f2e6b820ffd60aafe2"
server_key_iv: "9786e4b80b7f5e1bbaee6aadeb838814"
server_key_version: "encryptionkey@0"
I20260928 20:19:47.419339 8580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790626764250411-3144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790626764250411-3144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790626764250411-3144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:19:47.435356 8580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:19:47.435753 8580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:19:47.436028 8580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:19:47.439939 8580 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.62:39211
I20260928 20:19:47.439998 8646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.62:39211 every 8 connection(s)
I20260928 20:19:47.440389 8580 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
I20260928 20:19:47.441411 8647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:47.443604 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 8580
I20260928 20:19:47.443738 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790626764250411-3144-0/minicluster-data/master-0/wal/instance
I20260928 20:19:47.443972 3144 external_mini_cluster.cc:1428] Setting key 5a6f384dda9f5ad8cc920ad5fc2085c8
I20260928 20:19:47.444124 8647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2ce021e04cb407590760f7777224f8b: Bootstrap starting.
I20260928 20:19:47.444837 8647 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2ce021e04cb407590760f7777224f8b: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:47.445161 8647 log.cc:824] T 00000000000000000000000000000000 P a2ce021e04cb407590760f7777224f8b: Log is configured to *not* fsync() on all Append() calls
I20260928 20:19:47.445879 8647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2ce021e04cb407590760f7777224f8b: No bootstrap required, opened a new log
I20260928 20:19:47.447836 8647 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2ce021e04cb407590760f7777224f8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2ce021e04cb407590760f7777224f8b" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 39211 } }
I20260928 20:19:47.447993 8647 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2ce021e04cb407590760f7777224f8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:47.448019 8647 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2ce021e04cb407590760f7777224f8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2ce021e04cb407590760f7777224f8b, State: Initialized, Role: FOLLOWER
I20260928 20:19:47.448171 8647 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2ce021e04cb407590760f7777224f8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2ce021e04cb407590760f7777224f8b" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 39211 } }
I20260928 20:19:47.448252 8647 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2ce021e04cb407590760f7777224f8b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:19:47.448275 8647 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2ce021e04cb407590760f7777224f8b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:19:47.448340 8647 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2ce021e04cb407590760f7777224f8b [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:47.449167 8647 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2ce021e04cb407590760f7777224f8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2ce021e04cb407590760f7777224f8b" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 39211 } }
I20260928 20:19:47.449303 8647 leader_election.cc:302] T 00000000000000000000000000000000 P a2ce021e04cb407590760f7777224f8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2ce021e04cb407590760f7777224f8b; no voters:
I20260928 20:19:47.449546 8647 leader_election.cc:288] T 00000000000000000000000000000000 P a2ce021e04cb407590760f7777224f8b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:19:47.449568 8652 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2ce021e04cb407590760f7777224f8b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:19:47.449687 8652 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2ce021e04cb407590760f7777224f8b [term 1 LEADER]: Becoming Leader. State: Replica: a2ce021e04cb407590760f7777224f8b, State: Running, Role: LEADER
I20260928 20:19:47.449779 8652 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2ce021e04cb407590760f7777224f8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2ce021e04cb407590760f7777224f8b" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 39211 } }
I20260928 20:19:47.449854 8647 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2ce021e04cb407590760f7777224f8b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:19:47.450150 8654 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2ce021e04cb407590760f7777224f8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2ce021e04cb407590760f7777224f8b. Latest consensus state: current_term: 1 leader_uuid: "a2ce021e04cb407590760f7777224f8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2ce021e04cb407590760f7777224f8b" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 39211 } } }
I20260928 20:19:47.450277 8654 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2ce021e04cb407590760f7777224f8b [sys.catalog]: This master's current role is: LEADER
I20260928 20:19:47.450469 8650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:47.444554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38452 (local address 127.3.18.62:39211)
0928 20:19:47.444779 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:47.444787 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:47.444813 (+ 26us) server_negotiation.cc:407] Connection header received
0928 20:19:47.445394 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:47.445404 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:47.445482 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:47.445636 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:47.445818 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:47.446793 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:47.447644 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:47.448238 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:47.448394 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:47.449324 (+ 930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:47.449330 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:47.449330 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:19:47.449544 (+ 214us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:47.449580 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:19:47.449614 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:47.450023 (+ 409us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:47.450055 (+ 32us) server_negotiation.cc:299] Negotiation successful
0928 20:19:47.450156 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":144,"thread_start_us":87,"threads_started":1}
I20260928 20:19:47.450513 8653 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2ce021e04cb407590760f7777224f8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2ce021e04cb407590760f7777224f8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2ce021e04cb407590760f7777224f8b" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 39211 } } }
I20260928 20:19:47.450563 8653 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2ce021e04cb407590760f7777224f8b [sys.catalog]: This master's current role is: LEADER
I20260928 20:19:47.450713 8658 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:19:47.451264 8658 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:19:47.451704 3144 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 20:19:47.453297 8658 catalog_manager.cc:1347] Generated new cluster ID: b7971874eb354e32a93dbb4f87845e0a
I20260928 20:19:47.453420 8658 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:19:47.464015 8658 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:19:47.464562 8658 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:19:47.470239 8658 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2ce021e04cb407590760f7777224f8b: Generated new TSK 0
I20260928 20:19:47.470530 8658 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 20:19:47.616645 8650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:47.484258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38466 (local address 127.3.18.62:39211)
0928 20:19:47.484426 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:47.484429 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:47.484442 (+ 13us) server_negotiation.cc:407] Connection header received
0928 20:19:47.484495 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:47.484496 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:47.484506 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:47.484574 (+ 68us) server_negotiation.cc:226] Negotiated authn=JWT
0928 20:19:47.484729 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:47.485223 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:47.486377 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:47.486623 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:47.486687 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:47.486886 (+ 199us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 20:19:47.616408 (+129522us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 20:19:47.616582 (+ 174us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38466: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":122}
W20260928 20:19:47.616714 8650 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38466: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260928 20:19:47.616819 8676 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:19:47.484166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:39211 (local address 127.0.0.1:38466)
0928 20:19:47.484355 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:47.484368 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:47.484427 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:47.484598 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:47.484599 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:47.484601 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 20:19:47.484665 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:47.484667 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:47.485316 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:47.485317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:47.486258 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:47.486261 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:47.486709 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:47.486718 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:47.486779 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:47.486785 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 20:19:47.616563 (+129778us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 20:19:47.616686 (+ 123us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.18.62:39211: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":101,"threads_started":1}
I20260928 20:19:47.617223 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 8580
2026-09-28T20:19:47Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (334 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (89 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (93 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:19:47 dist-test-slave-b83v krb5kdc[8688](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:19:47 dist-test-slave-b83v krb5kdc[8688](info): set up 2 sockets
Sep 28 20:19:47 dist-test-slave-b83v krb5kdc[8688](info): commencing operation
krb5kdc: starting...
W20260928 20:19:48.856310 3144 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Sep 28 20:19:48 dist-test-slave-b83v krb5kdc[8688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:19:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:19:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.18.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab.
Entry for principal kudu/127.3.18.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab.
I20260928 20:19:48.987885 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:36795
--webserver_interface=127.3.18.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:46143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.18.62:36795
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:19:49.069352 8701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:19:49.069589 8701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:19:49.069614 8701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:19:49.070749 8701 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:19:49.070811 8701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:19:49.070827 8701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:19:49.070839 8701 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:19:49.070851 8701 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:19:49.072337 8701 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:46143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.18.62:36795
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:36795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.18.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:19:49.072548 8701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:19:49.072726 8701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:19:49.078271 8707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:49.078287 8706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:49.078397 8709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:49.078734 8701 server_base.cc:1034] running on GCE node
I20260928 20:19:49.078971 8701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:19:49.079233 8701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:19:49.080379 8701 hybrid_clock.cc:648] HybridClock initialized: now 1790626789080365 us; error 33 us; skew 500 ppm
Sep 28 20:19:49 dist-test-slave-b83v krb5kdc[8688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626789, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:19:49.083201 8701 init.cc:377] Logged in from keytab as kudu/127.3.18.62@KRBTEST.COM (short username kudu)
I20260928 20:19:49.083691 8701 webserver.cc:460] Webserver started at http://127.3.18.62:33377/ using document root <none> and password file <none>
I20260928 20:19:49.083974 8701 fs_manager.cc:359] Metadata directory not provided
I20260928 20:19:49.084023 8701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:19:49.084188 8701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:19:49.085243 8701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/minicluster-data/master-0/data/instance:
uuid: "b2280aa834b64863a62f78c01d03ab98"
format_stamp: "Formatted at 2026-09-28 20:19:49 on dist-test-slave-b83v"
server_key: "5e62c716b480252fdaa2319bfd7193e8"
server_key_iv: "a64a4c74df13193f44631d9320be4571"
server_key_version: "encryptionkey@0"
I20260928 20:19:49.085602 8701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/minicluster-data/master-0/wal/instance:
uuid: "b2280aa834b64863a62f78c01d03ab98"
format_stamp: "Formatted at 2026-09-28 20:19:49 on dist-test-slave-b83v"
server_key: "5e62c716b480252fdaa2319bfd7193e8"
server_key_iv: "a64a4c74df13193f44631d9320be4571"
server_key_version: "encryptionkey@0"
I20260928 20:19:49.087318 8701 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 20:19:49.088315 8716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:49.088593 8701 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 20:19:49.088706 8701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/minicluster-data/master-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/minicluster-data/master-0/wal
uuid: "b2280aa834b64863a62f78c01d03ab98"
format_stamp: "Formatted at 2026-09-28 20:19:49 on dist-test-slave-b83v"
server_key: "5e62c716b480252fdaa2319bfd7193e8"
server_key_iv: "a64a4c74df13193f44631d9320be4571"
server_key_version: "encryptionkey@0"
I20260928 20:19:49.088790 8701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:19:49.120357 8701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:19:49.122867 8701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:19:49.123032 8701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:19:49.127254 8701 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.62:36795
I20260928 20:19:49.127310 8768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.62:36795 every 8 connection(s)
I20260928 20:19:49.127707 8701 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
I20260928 20:19:49.128729 8769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:49.131217 8769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2280aa834b64863a62f78c01d03ab98: Bootstrap starting.
I20260928 20:19:49.131837 8769 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2280aa834b64863a62f78c01d03ab98: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:49.132124 8769 log.cc:824] T 00000000000000000000000000000000 P b2280aa834b64863a62f78c01d03ab98: Log is configured to *not* fsync() on all Append() calls
I20260928 20:19:49.132877 8769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2280aa834b64863a62f78c01d03ab98: No bootstrap required, opened a new log
I20260928 20:19:49.133396 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 8701
I20260928 20:19:49.133618 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/minicluster-data/master-0/wal/instance
I20260928 20:19:49.133854 3144 external_mini_cluster.cc:1428] Setting key 7448ed3c9eaa0f05f0881bb1d75bb9c2
I20260928 20:19:49.135377 8769 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2280aa834b64863a62f78c01d03ab98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2280aa834b64863a62f78c01d03ab98" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36795 } }
I20260928 20:19:49.135583 8769 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2280aa834b64863a62f78c01d03ab98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:49.135645 8769 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2280aa834b64863a62f78c01d03ab98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2280aa834b64863a62f78c01d03ab98, State: Initialized, Role: FOLLOWER
I20260928 20:19:49.135876 8769 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2280aa834b64863a62f78c01d03ab98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2280aa834b64863a62f78c01d03ab98" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36795 } }
I20260928 20:19:49.136022 8769 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2280aa834b64863a62f78c01d03ab98 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:19:49.136063 8769 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2280aa834b64863a62f78c01d03ab98 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:19:49.136124 8769 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2280aa834b64863a62f78c01d03ab98 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:49.137297 8769 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2280aa834b64863a62f78c01d03ab98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2280aa834b64863a62f78c01d03ab98" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36795 } }
I20260928 20:19:49.137394 8769 leader_election.cc:302] T 00000000000000000000000000000000 P b2280aa834b64863a62f78c01d03ab98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2280aa834b64863a62f78c01d03ab98; no voters:
I20260928 20:19:49.137627 8769 leader_election.cc:288] T 00000000000000000000000000000000 P b2280aa834b64863a62f78c01d03ab98 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:19:49.137677 8774 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2280aa834b64863a62f78c01d03ab98 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:19:49.137835 8774 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2280aa834b64863a62f78c01d03ab98 [term 1 LEADER]: Becoming Leader. State: Replica: b2280aa834b64863a62f78c01d03ab98, State: Running, Role: LEADER
I20260928 20:19:49.137941 8769 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2280aa834b64863a62f78c01d03ab98 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:19:49.137953 8774 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2280aa834b64863a62f78c01d03ab98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2280aa834b64863a62f78c01d03ab98" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36795 } }
I20260928 20:19:49.138338 8776 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2280aa834b64863a62f78c01d03ab98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2280aa834b64863a62f78c01d03ab98. Latest consensus state: current_term: 1 leader_uuid: "b2280aa834b64863a62f78c01d03ab98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2280aa834b64863a62f78c01d03ab98" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36795 } } }
I20260928 20:19:49.138474 8776 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2280aa834b64863a62f78c01d03ab98 [sys.catalog]: This master's current role is: LEADER
I20260928 20:19:49.138648 8775 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2280aa834b64863a62f78c01d03ab98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2280aa834b64863a62f78c01d03ab98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2280aa834b64863a62f78c01d03ab98" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36795 } } }
I20260928 20:19:49.138715 8775 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2280aa834b64863a62f78c01d03ab98 [sys.catalog]: This master's current role is: LEADER
I20260928 20:19:49.139074 8781 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:19:49.139951 8781 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:19:49.141758 8781 catalog_manager.cc:1347] Generated new cluster ID: f0f3e2f0f13d45a5b0778d1c2790713e
I20260928 20:19:49.141810 8781 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 20:19:49 dist-test-slave-b83v krb5kdc[8688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:19:49.147317 8781 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:19:49.147471 8773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:49.134452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37698 (local address 127.3.18.62:36795)
0928 20:19:49.134647 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:49.134656 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:49.134682 (+ 26us) server_negotiation.cc:407] Connection header received
0928 20:19:49.135348 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:49.135354 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:49.135404 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:49.135524 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:49.136558 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:49.137647 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:49.139265 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:49.139961 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:49.140052 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:49.143234 (+ 3182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:49.143239 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:49.143239 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:49.143251 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:49.145190 (+ 1939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:49.145652 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:49.145655 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:49.145655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:49.146134 (+ 479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:49.146386 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:49.146387 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:49.146388 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:49.146685 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:49.146835 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:49.146995 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 20:19:49.147175 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":113,"threads_started":1}
I20260928 20:19:49.148159 8781 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:19:49.153121 8781 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2280aa834b64863a62f78c01d03ab98: Generated new TSK 0
I20260928 20:19:49.153326 8781 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.18.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab.
Entry for principal kudu/127.3.18.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab.
I20260928 20:19:49.189796 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.1:0
--local_ip_for_outbound_sockets=127.3.18.1
--webserver_interface=127.3.18.1
--webserver_port=0
--tserver_master_addrs=127.3.18.62:36795
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab
--principal=kudu/127.3.18.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:46143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:19:49.276744 8795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:19:49.276968 8795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:19:49.276994 8795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:19:49.278221 8795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:19:49.278272 8795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:19:49.278301 8795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.1
I20260928 20:19:49.279996 8795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:46143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab
--principal=kudu/127.3.18.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.18.1
--webserver_port=0
--tserver_master_addrs=127.3.18.62:36795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.1
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:19:49.280318 8795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:19:49.280578 8795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:19:49.286239 8801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:49.286296 8795 server_base.cc:1034] running on GCE node
W20260928 20:19:49.286237 8800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:49.286288 8803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:49.286732 8795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:19:49.287036 8795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:19:49.288210 8795 hybrid_clock.cc:648] HybridClock initialized: now 1790626789288194 us; error 49 us; skew 500 ppm
Sep 28 20:19:49 dist-test-slave-b83v krb5kdc[8688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626789, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:19:49.291096 8795 init.cc:377] Logged in from keytab as kudu/127.3.18.1@KRBTEST.COM (short username kudu)
I20260928 20:19:49.291687 8795 webserver.cc:460] Webserver started at http://127.3.18.1:44299/ using document root <none> and password file <none>
I20260928 20:19:49.291991 8795 fs_manager.cc:359] Metadata directory not provided
I20260928 20:19:49.292071 8795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:19:49.292227 8795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:19:49.293499 8795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/minicluster-data/ts-0/data/instance:
uuid: "7f06a6919b664644abdff8e1024302fb"
format_stamp: "Formatted at 2026-09-28 20:19:49 on dist-test-slave-b83v"
server_key: "0260474dc6cbdaae6701ba5e5ebac5e9"
server_key_iv: "69e314e913de009de184f40d195e4bcb"
server_key_version: "encryptionkey@0"
I20260928 20:19:49.293901 8795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/minicluster-data/ts-0/wal/instance:
uuid: "7f06a6919b664644abdff8e1024302fb"
format_stamp: "Formatted at 2026-09-28 20:19:49 on dist-test-slave-b83v"
server_key: "0260474dc6cbdaae6701ba5e5ebac5e9"
server_key_iv: "69e314e913de009de184f40d195e4bcb"
server_key_version: "encryptionkey@0"
I20260928 20:19:49.295331 8795 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:19:49.296379 8810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:49.296636 8795 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 20:19:49.296746 8795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/minicluster-data/ts-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/minicluster-data/ts-0/wal
uuid: "7f06a6919b664644abdff8e1024302fb"
format_stamp: "Formatted at 2026-09-28 20:19:49 on dist-test-slave-b83v"
server_key: "0260474dc6cbdaae6701ba5e5ebac5e9"
server_key_iv: "69e314e913de009de184f40d195e4bcb"
server_key_version: "encryptionkey@0"
I20260928 20:19:49.296836 8795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:19:49.309403 8795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:19:49.311626 8795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:19:49.311851 8795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:19:49.312135 8795 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:19:49.312498 8795 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:19:49.312556 8795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:49.312600 8795 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:19:49.312623 8795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:49.318809 8795 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.1:38573
I20260928 20:19:49.318886 8923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.1:38573 every 8 connection(s)
I20260928 20:19:49.319231 8795 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
I20260928 20:19:49.325783 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 8795
I20260928 20:19:49.325881 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/minicluster-data/ts-0/wal/instance
Sep 28 20:19:49 dist-test-slave-b83v krb5kdc[8688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626789, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.1@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:19:49.326143 3144 external_mini_cluster.cc:1428] Setting key 284a6d67ece1f0844d2b90747490efc3
I20260928 20:19:49.330586 8773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:49.320115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:37331 (local address 127.3.18.62:36795)
0928 20:19:49.320183 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:49.320185 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:49.320402 (+ 217us) server_negotiation.cc:407] Connection header received
0928 20:19:49.321456 (+ 1054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:49.321458 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:49.321476 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:49.321534 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:49.322219 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:49.322745 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:49.323815 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:49.324028 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:49.324107 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:49.326455 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:49.326456 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:49.326457 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:49.326496 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:49.328441 (+ 1945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:49.329011 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:49.329012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:49.329013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:49.329093 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:49.329656 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:49.329657 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:49.329658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:49.329847 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:49.329916 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:49.330413 (+ 497us) server_negotiation.cc:299] Negotiation successful
0928 20:19:49.330513 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 20:19:49.330922 8926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:49.320095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36795 (local address 127.3.18.1:37331)
0928 20:19:49.320273 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:49.320310 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:49.321323 (+ 1013us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:49.321616 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:49.321620 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:49.321889 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:49.322117 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:49.322121 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:49.322858 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:49.322859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:49.323685 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:49.323687 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:49.324115 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:49.324116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:49.324250 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:49.324825 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:19:49.324835 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:49.326273 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:49.328584 (+ 2311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:49.328594 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:49.328596 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:49.328901 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:49.329192 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:49.329193 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:49.329194 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:49.329529 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:49.329987 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:49.329995 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:49.330216 (+ 221us) client_negotiation.cc:769] Sending connection context
0928 20:19:49.330358 (+ 142us) client_negotiation.cc:240] Negotiation successful
0928 20:19:49.330594 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":79,"threads_started":1}
I20260928 20:19:49.331744 8924 heartbeater.cc:347] Connected to a master server at 127.3.18.62:36795
I20260928 20:19:49.331918 8924 heartbeater.cc:464] Registering TS with master...
I20260928 20:19:49.332166 8924 heartbeater.cc:511] Master 127.3.18.62:36795 requested a full tablet report, sending...
I20260928 20:19:49.332799 8733 ts_manager.cc:194] Registered new tserver with Master: 7f06a6919b664644abdff8e1024302fb (127.3.18.1:38573)
I20260928 20:19:49.333496 8733 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.18.1@KRBTEST.COM'} at 127.3.18.1:37331
WARNING: no policy specified for kudu/127.3.18.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab.
Entry for principal kudu/127.3.18.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab.
I20260928 20:19:49.361285 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.2:0
--local_ip_for_outbound_sockets=127.3.18.2
--webserver_interface=127.3.18.2
--webserver_port=0
--tserver_master_addrs=127.3.18.62:36795
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab
--principal=kudu/127.3.18.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:46143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:19:49.444252 8929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:19:49.444535 8929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:19:49.444576 8929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:19:49.445823 8929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:19:49.445907 8929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:19:49.445955 8929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.2
I20260928 20:19:49.447566 8929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:46143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab
--principal=kudu/127.3.18.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.18.2
--webserver_port=0
--tserver_master_addrs=127.3.18.62:36795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.2
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:19:49.447909 8929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:19:49.448160 8929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:19:49.453598 8934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:49.453589 8937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:49.453689 8929 server_base.cc:1034] running on GCE node
W20260928 20:19:49.453589 8935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:49.453996 8929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:19:49.454231 8929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:19:49.455401 8929 hybrid_clock.cc:648] HybridClock initialized: now 1790626789455379 us; error 50 us; skew 500 ppm
Sep 28 20:19:49 dist-test-slave-b83v krb5kdc[8688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626789, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:19:49.458153 8929 init.cc:377] Logged in from keytab as kudu/127.3.18.2@KRBTEST.COM (short username kudu)
I20260928 20:19:49.458716 8929 webserver.cc:460] Webserver started at http://127.3.18.2:40755/ using document root <none> and password file <none>
I20260928 20:19:49.458978 8929 fs_manager.cc:359] Metadata directory not provided
I20260928 20:19:49.459050 8929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:19:49.459204 8929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:19:49.460353 8929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/minicluster-data/ts-1/data/instance:
uuid: "78c0ed17de0448e5a090e6de182b915c"
format_stamp: "Formatted at 2026-09-28 20:19:49 on dist-test-slave-b83v"
server_key: "b7317288a98e2c1a56872a1b7290aba8"
server_key_iv: "f3a19281be01398a700ae86b51bb4e63"
server_key_version: "encryptionkey@0"
I20260928 20:19:49.460731 8929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/minicluster-data/ts-1/wal/instance:
uuid: "78c0ed17de0448e5a090e6de182b915c"
format_stamp: "Formatted at 2026-09-28 20:19:49 on dist-test-slave-b83v"
server_key: "b7317288a98e2c1a56872a1b7290aba8"
server_key_iv: "f3a19281be01398a700ae86b51bb4e63"
server_key_version: "encryptionkey@0"
I20260928 20:19:49.462096 8929 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 20:19:49.462888 8944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:49.463114 8929 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:19:49.463219 8929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/minicluster-data/ts-1/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/minicluster-data/ts-1/wal
uuid: "78c0ed17de0448e5a090e6de182b915c"
format_stamp: "Formatted at 2026-09-28 20:19:49 on dist-test-slave-b83v"
server_key: "b7317288a98e2c1a56872a1b7290aba8"
server_key_iv: "f3a19281be01398a700ae86b51bb4e63"
server_key_version: "encryptionkey@0"
I20260928 20:19:49.463299 8929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:19:49.475512 8929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:19:49.477771 8929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:19:49.478025 8929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:19:49.478427 8929 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:19:49.478832 8929 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:19:49.478902 8929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:49.478951 8929 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:19:49.478983 8929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:49.486006 8929 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.2:33503
I20260928 20:19:49.486042 9057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.2:33503 every 8 connection(s)
I20260928 20:19:49.486457 8929 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
I20260928 20:19:49.486826 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 8929
I20260928 20:19:49.486948 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/minicluster-data/ts-1/wal/instance
I20260928 20:19:49.487226 3144 external_mini_cluster.cc:1428] Setting key 9d1b58a283a406307cad003158ba8182
Sep 28 20:19:49 dist-test-slave-b83v krb5kdc[8688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626789, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.2@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.18.3@KRBTEST.COM; defaulting to no policy
I20260928 20:19:49.497812 8773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:49.487553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:49085 (local address 127.3.18.62:36795)
0928 20:19:49.487653 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:49.487656 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:49.487971 (+ 315us) server_negotiation.cc:407] Connection header received
0928 20:19:49.489366 (+ 1395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:49.489373 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:49.489383 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:49.489435 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:49.490286 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:49.490822 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:49.492042 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:49.492380 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:49.492466 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:49.494709 (+ 2243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:49.494711 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:49.494711 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:49.494720 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:49.496340 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:49.496827 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:49.496829 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:49.496829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:49.496889 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:49.497142 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:49.497142 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:49.497143 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:49.497261 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:49.497344 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:49.497618 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 20:19:49.497716 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 20:19:49.498032 9060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:49.487550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36795 (local address 127.3.18.2:49085)
0928 20:19:49.487795 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:49.487846 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:49.489212 (+ 1366us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:49.489514 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:49.489519 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:49.489922 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:49.490156 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:49.490162 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:49.490945 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:49.490946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:49.491931 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:49.491934 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:49.492475 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:49.492476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:49.492610 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:49.493191 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:19:49.493200 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:49.494612 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:49.496494 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:49.496502 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:49.496504 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:49.496739 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:49.496960 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:49.496960 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:49.496961 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:49.497070 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:49.497326 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:49.497333 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:49.497484 (+ 151us) client_negotiation.cc:769] Sending connection context
0928 20:19:49.497598 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 20:19:49.497776 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":113,"threads_started":1}
I20260928 20:19:49.498570 9058 heartbeater.cc:347] Connected to a master server at 127.3.18.62:36795
I20260928 20:19:49.498703 9058 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.3@KRBTEST.COM" created.
I20260928 20:19:49.498945 9058 heartbeater.cc:511] Master 127.3.18.62:36795 requested a full tablet report, sending...
I20260928 20:19:49.499398 8733 ts_manager.cc:194] Registered new tserver with Master: 78c0ed17de0448e5a090e6de182b915c (127.3.18.2:33503)
I20260928 20:19:49.499907 8733 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.18.2@KRBTEST.COM'} at 127.3.18.2:49085
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab.
Entry for principal kudu/127.3.18.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab.
I20260928 20:19:49.516676 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.3:0
--local_ip_for_outbound_sockets=127.3.18.3
--webserver_interface=127.3.18.3
--webserver_port=0
--tserver_master_addrs=127.3.18.62:36795
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab
--principal=kudu/127.3.18.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:46143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:19:49.599237 9063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:19:49.599471 9063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:19:49.599509 9063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:19:49.600751 9063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:19:49.600795 9063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:19:49.600821 9063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.3
I20260928 20:19:49.602283 9063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:46143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab
--principal=kudu/127.3.18.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.18.3
--webserver_port=0
--tserver_master_addrs=127.3.18.62:36795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.3
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:19:49.602550 9063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:19:49.602782 9063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 20:19:49.608534 9063 server_base.cc:1034] running on GCE node
W20260928 20:19:49.608506 9071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:49.608569 9068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:49.608510 9069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:49.609128 9063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:19:49.609458 9063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:19:49.610613 9063 hybrid_clock.cc:648] HybridClock initialized: now 1790626789610599 us; error 30 us; skew 500 ppm
Sep 28 20:19:49 dist-test-slave-b83v krb5kdc[8688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626789, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:19:49.613495 9063 init.cc:377] Logged in from keytab as kudu/127.3.18.3@KRBTEST.COM (short username kudu)
I20260928 20:19:49.614018 9063 webserver.cc:460] Webserver started at http://127.3.18.3:43887/ using document root <none> and password file <none>
I20260928 20:19:49.614272 9063 fs_manager.cc:359] Metadata directory not provided
I20260928 20:19:49.614346 9063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:19:49.614511 9063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:19:49.615622 9063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/minicluster-data/ts-2/data/instance:
uuid: "de759182de5f4b97bac3abc3f5203819"
format_stamp: "Formatted at 2026-09-28 20:19:49 on dist-test-slave-b83v"
server_key: "3cdce8593c20d883dc54a0ce02596757"
server_key_iv: "3086818fe9c26c45668828b44e9a9d17"
server_key_version: "encryptionkey@0"
I20260928 20:19:49.616037 9063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/minicluster-data/ts-2/wal/instance:
uuid: "de759182de5f4b97bac3abc3f5203819"
format_stamp: "Formatted at 2026-09-28 20:19:49 on dist-test-slave-b83v"
server_key: "3cdce8593c20d883dc54a0ce02596757"
server_key_iv: "3086818fe9c26c45668828b44e9a9d17"
server_key_version: "encryptionkey@0"
I20260928 20:19:49.617470 9063 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:19:49.618286 9078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:49.618516 9063 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:19:49.618649 9063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/minicluster-data/ts-2/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/minicluster-data/ts-2/wal
uuid: "de759182de5f4b97bac3abc3f5203819"
format_stamp: "Formatted at 2026-09-28 20:19:49 on dist-test-slave-b83v"
server_key: "3cdce8593c20d883dc54a0ce02596757"
server_key_iv: "3086818fe9c26c45668828b44e9a9d17"
server_key_version: "encryptionkey@0"
I20260928 20:19:49.618741 9063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:19:49.635308 9063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:19:49.637503 9063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:19:49.637678 9063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:19:49.637920 9063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:19:49.638427 9063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:19:49.638487 9063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:49.638533 9063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:19:49.638562 9063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:49.645278 9063 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.3:36275
I20260928 20:19:49.645328 9191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.3:36275 every 8 connection(s)
I20260928 20:19:49.645670 9063 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
I20260928 20:19:49.652413 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 9063
I20260928 20:19:49.652493 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/minicluster-data/ts-2/wal/instance
Sep 28 20:19:49 dist-test-slave-b83v krb5kdc[8688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626789, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.3@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:19:49.652705 3144 external_mini_cluster.cc:1428] Setting key 16f6c273160af2a9f67e8ae428734d7d
I20260928 20:19:49.655911 8773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:49.646530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:60229 (local address 127.3.18.62:36795)
0928 20:19:49.646586 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:49.646588 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:49.646761 (+ 173us) server_negotiation.cc:407] Connection header received
0928 20:19:49.648110 (+ 1349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:49.648112 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:49.648124 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:49.648186 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:49.648824 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:49.649320 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:49.650510 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:49.650700 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:49.650757 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:49.653159 (+ 2402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:49.653161 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:49.653161 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:49.653170 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:49.654657 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:49.655021 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:49.655022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:49.655022 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:49.655072 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:49.655284 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:49.655285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:49.655285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:49.655434 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:49.655486 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:49.655716 (+ 230us) server_negotiation.cc:299] Negotiation successful
0928 20:19:49.655831 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 20:19:49.656091 9194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:49.646429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36795 (local address 127.3.18.3:60229)
0928 20:19:49.646615 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:49.646661 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:49.647977 (+ 1316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:49.648247 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:49.648251 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:49.648532 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:49.648740 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:49.648745 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:49.649470 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:49.649471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:49.650381 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:49.650383 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:49.650781 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:49.650782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:49.650896 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:49.651370 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:19:49.651379 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:49.653063 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:49.654728 (+ 1665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:49.654734 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:49.654736 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:49.654957 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:49.655128 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:49.655129 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:49.655129 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:49.655232 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:49.655491 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:49.655495 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:49.655609 (+ 114us) client_negotiation.cc:769] Sending connection context
0928 20:19:49.655689 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 20:19:49.655871 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":99,"threads_started":1}
I20260928 20:19:49.656567 9192 heartbeater.cc:347] Connected to a master server at 127.3.18.62:36795
I20260928 20:19:49.656647 9192 heartbeater.cc:464] Registering TS with master...
I20260928 20:19:49.656788 9192 heartbeater.cc:511] Master 127.3.18.62:36795 requested a full tablet report, sending...
I20260928 20:19:49.657131 8733 ts_manager.cc:194] Registered new tserver with Master: de759182de5f4b97bac3abc3f5203819 (127.3.18.3:36275)
I20260928 20:19:49.657567 8733 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.18.3@KRBTEST.COM'} at 127.3.18.3:60229
I20260928 20:19:49.664655 3144 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/krb5kdc/test-admin.keytab.
Sep 28 20:19:49 dist-test-slave-b83v krb5kdc[8688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:19:49.681632 3144 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 20:19:49.683427 9199 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:19:49.682595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47034 (local address 127.3.18.1:38573)
0928 20:19:49.682755 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:49.682761 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:49.682773 (+ 12us) server_negotiation.cc:407] Connection header received
0928 20:19:49.682810 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:49.682812 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:49.682826 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:49.682902 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:49.683276 (+ 374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47034: BlockingRecv error: recv got EOF from 127.0.0.1:47034 (error 108)
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":90,"threads_started":1}
W20260928 20:19:49.683512 3144 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/krb5kdc/krb5cc' of type FILE
Sep 28 20:19:49 dist-test-slave-b83v krb5kdc[8688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:19:49.684656 3144 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 20:19:49 dist-test-slave-b83v krb5kdc[8688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:19:49.685627 3144 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 20:19:49 dist-test-slave-b83v krb5kdc[8688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.18.1@KRBTEST.COM
I20260928 20:19:49.694936 9199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:49.686277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47046 (local address 127.3.18.1:38573)
0928 20:19:49.686352 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:49.686353 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:49.686460 (+ 107us) server_negotiation.cc:407] Connection header received
0928 20:19:49.686485 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:49.686486 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:49.686499 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:49.686541 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:49.687026 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:49.688022 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:49.688797 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:49.689152 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:49.689211 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:49.691800 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:49.691812 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:49.691813 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:49.691823 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:49.693787 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:49.694158 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:49.694160 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:49.694161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:49.694214 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:49.694379 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:49.694380 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:49.694380 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:49.694519 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:49.694639 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:49.694677 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 20:19:49.694739 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260928 20:19:49.697189 9199 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:19:49.696476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47048 (local address 127.3.18.1:38573)
0928 20:19:49.696566 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:49.696568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:49.696634 (+ 66us) server_negotiation.cc:407] Connection header received
0928 20:19:49.696713 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:49.696713 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:49.696725 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:49.696801 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:49.697118 (+ 317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47048: BlockingRecv error: recv got EOF from 127.0.0.1:47048 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260928 20:19:49.697306 3144 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/krb5kdc/krb5cc' of type FILE
Sep 28 20:19:49 dist-test-slave-b83v krb5kdc[8688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:19:49.698385 3144 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 20:19:49 dist-test-slave-b83v krb5kdc[8688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:19:49.699404 3144 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 20:19:49 dist-test-slave-b83v krb5kdc[8688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.18.1@KRBTEST.COM
I20260928 20:19:49.707559 9199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:49.700026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47050 (local address 127.3.18.1:38573)
0928 20:19:49.700062 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:49.700064 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:49.700178 (+ 114us) server_negotiation.cc:407] Connection header received
0928 20:19:49.700253 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:49.700254 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:49.700266 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:49.700309 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:49.700789 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:49.701304 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:49.702036 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:49.702250 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:49.702351 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:49.704742 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:49.704744 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:49.704744 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:49.704752 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:49.706288 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:49.706667 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:49.706668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:49.706668 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:49.706730 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:49.706994 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:49.706995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:49.706995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:49.707111 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:49.707195 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:49.707359 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 20:19:49.707485 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
W20260928 20:19:49.709759 9199 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:19:49.708892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47058 (local address 127.3.18.1:38573)
0928 20:19:49.709001 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:49.709002 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:49.709011 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:19:49.709062 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:49.709062 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:49.709071 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:49.709128 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:49.709685 (+ 557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47058: BlockingRecv error: recv got EOF from 127.0.0.1:47058 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
W20260928 20:19:49.709796 3144 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790626764250411-3144-0/krb5kdc/krb5cc' of type FILE
Sep 28 20:19:49 dist-test-slave-b83v krb5kdc[8688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:19:49.710983 3144 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 20:19:49 dist-test-slave-b83v krb5kdc[8688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:19:49.712061 3144 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 20:19:49 dist-test-slave-b83v krb5kdc[8688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.18.1@KRBTEST.COM
I20260928 20:19:49.720330 9199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:49.712660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47074 (local address 127.3.18.1:38573)
0928 20:19:49.712727 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:49.712730 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:49.712798 (+ 68us) server_negotiation.cc:407] Connection header received
0928 20:19:49.712895 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:49.712896 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:49.712916 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:49.713042 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:49.713521 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:49.714035 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:49.714838 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:49.715079 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:49.715164 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:49.717585 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:49.717587 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:49.717587 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:49.717603 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:49.719035 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:49.719431 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:49.719433 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:49.719433 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:49.719496 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:49.719710 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:49.719711 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:49.719712 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:49.719887 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:49.720023 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:49.720128 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 20:19:49.720217 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 20:19:49.720974 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 8795
I20260928 20:19:49.726053 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 8929
I20260928 20:19:49.730476 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 9063
I20260928 20:19:49.735074 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 8701
2026-09-28T20:19:49Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1936 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:19:49 dist-test-slave-b83v krb5kdc[9212](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:19:49 dist-test-slave-b83v krb5kdc[9212](info): set up 2 sockets
Sep 28 20:19:49 dist-test-slave-b83v krb5kdc[9212](info): commencing operation
krb5kdc: starting...
W20260928 20:19:50.800253 3144 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Sep 28 20:19:50 dist-test-slave-b83v krb5kdc[9212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:19:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:19:50Z Disabled control of system clock
WARNING: no policy specified for oryx/127.3.18.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.18.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.18.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/krb5kdc/oryx_127.3.18.62.keytab.
Entry for principal oryx/127.3.18.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/krb5kdc/oryx_127.3.18.62.keytab.
I20260928 20:19:50.927958 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:42379
--webserver_interface=127.3.18.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/krb5kdc/oryx_127.3.18.62.keytab
--principal=oryx/127.3.18.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:45321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.18.62:42379
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:19:51.014537 9225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:19:51.014890 9225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:19:51.014930 9225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:19:51.016285 9225 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:19:51.016331 9225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:19:51.016346 9225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:19:51.016359 9225 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:19:51.016371 9225 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:19:51.018072 9225 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:45321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.18.62:42379
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/krb5kdc/oryx_127.3.18.62.keytab
--principal=oryx/127.3.18.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:42379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.18.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9225
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:19:51.018289 9225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:19:51.018492 9225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:19:51.023869 9230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:51.023919 9231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:51.023952 9225 server_base.cc:1034] running on GCE node
W20260928 20:19:51.023931 9233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:51.024362 9225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:19:51.024731 9225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:19:51.025955 9225 hybrid_clock.cc:648] HybridClock initialized: now 1790626791025932 us; error 40 us; skew 500 ppm
Sep 28 20:19:51 dist-test-slave-b83v krb5kdc[9212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626791, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.18.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:19:51.029067 9225 init.cc:377] Logged in from keytab as oryx/127.3.18.62@KRBTEST.COM (short username oryx)
I20260928 20:19:51.029647 9225 webserver.cc:460] Webserver started at http://127.3.18.62:40725/ using document root <none> and password file <none>
I20260928 20:19:51.029917 9225 fs_manager.cc:359] Metadata directory not provided
I20260928 20:19:51.029989 9225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:19:51.030141 9225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:19:51.031275 9225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/master-0/data/instance:
uuid: "ef4554cccf754a53a7d74f685421731a"
format_stamp: "Formatted at 2026-09-28 20:19:51 on dist-test-slave-b83v"
server_key: "6dccba6633aecd036fc86dfae7845838"
server_key_iv: "7c5bff3a4abcf8b03db45cd2a42c843b"
server_key_version: "encryptionkey@0"
I20260928 20:19:51.031656 9225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/master-0/wal/instance:
uuid: "ef4554cccf754a53a7d74f685421731a"
format_stamp: "Formatted at 2026-09-28 20:19:51 on dist-test-slave-b83v"
server_key: "6dccba6633aecd036fc86dfae7845838"
server_key_iv: "7c5bff3a4abcf8b03db45cd2a42c843b"
server_key_version: "encryptionkey@0"
I20260928 20:19:51.033090 9225 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:19:51.034016 9240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:51.034245 9225 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:19:51.034363 9225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/master-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/master-0/wal
uuid: "ef4554cccf754a53a7d74f685421731a"
format_stamp: "Formatted at 2026-09-28 20:19:51 on dist-test-slave-b83v"
server_key: "6dccba6633aecd036fc86dfae7845838"
server_key_iv: "7c5bff3a4abcf8b03db45cd2a42c843b"
server_key_version: "encryptionkey@0"
I20260928 20:19:51.034432 9225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:19:51.046316 9225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:19:51.048764 9225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:19:51.049008 9225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:19:51.053669 9225 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.62:42379
I20260928 20:19:51.053773 9292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.62:42379 every 8 connection(s)
I20260928 20:19:51.054152 9225 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
I20260928 20:19:51.055328 9293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:51.057780 9293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef4554cccf754a53a7d74f685421731a: Bootstrap starting.
I20260928 20:19:51.058386 9293 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef4554cccf754a53a7d74f685421731a: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:51.058681 9293 log.cc:824] T 00000000000000000000000000000000 P ef4554cccf754a53a7d74f685421731a: Log is configured to *not* fsync() on all Append() calls
I20260928 20:19:51.059496 9293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef4554cccf754a53a7d74f685421731a: No bootstrap required, opened a new log
I20260928 20:19:51.061162 9293 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef4554cccf754a53a7d74f685421731a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef4554cccf754a53a7d74f685421731a" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 42379 } }
I20260928 20:19:51.061303 9293 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef4554cccf754a53a7d74f685421731a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:51.061326 9293 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef4554cccf754a53a7d74f685421731a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef4554cccf754a53a7d74f685421731a, State: Initialized, Role: FOLLOWER
I20260928 20:19:51.061529 9293 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef4554cccf754a53a7d74f685421731a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef4554cccf754a53a7d74f685421731a" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 42379 } }
I20260928 20:19:51.061587 9293 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef4554cccf754a53a7d74f685421731a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:19:51.061609 9293 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef4554cccf754a53a7d74f685421731a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:19:51.061636 9293 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef4554cccf754a53a7d74f685421731a [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:51.062474 9293 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef4554cccf754a53a7d74f685421731a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef4554cccf754a53a7d74f685421731a" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 42379 } }
I20260928 20:19:51.062566 9293 leader_election.cc:302] T 00000000000000000000000000000000 P ef4554cccf754a53a7d74f685421731a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef4554cccf754a53a7d74f685421731a; no voters:
I20260928 20:19:51.062747 9293 leader_election.cc:288] T 00000000000000000000000000000000 P ef4554cccf754a53a7d74f685421731a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:19:51.062844 9296 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef4554cccf754a53a7d74f685421731a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:19:51.063027 9296 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef4554cccf754a53a7d74f685421731a [term 1 LEADER]: Becoming Leader. State: Replica: ef4554cccf754a53a7d74f685421731a, State: Running, Role: LEADER
I20260928 20:19:51.063076 9293 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef4554cccf754a53a7d74f685421731a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:19:51.063124 9296 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef4554cccf754a53a7d74f685421731a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef4554cccf754a53a7d74f685421731a" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 42379 } }
I20260928 20:19:51.063585 9298 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef4554cccf754a53a7d74f685421731a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef4554cccf754a53a7d74f685421731a. Latest consensus state: current_term: 1 leader_uuid: "ef4554cccf754a53a7d74f685421731a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef4554cccf754a53a7d74f685421731a" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 42379 } } }
I20260928 20:19:51.063616 9297 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef4554cccf754a53a7d74f685421731a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef4554cccf754a53a7d74f685421731a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef4554cccf754a53a7d74f685421731a" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 42379 } } }
I20260928 20:19:51.063802 9297 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef4554cccf754a53a7d74f685421731a [sys.catalog]: This master's current role is: LEADER
I20260928 20:19:51.064033 9298 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef4554cccf754a53a7d74f685421731a [sys.catalog]: This master's current role is: LEADER
I20260928 20:19:51.064018 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 9225
I20260928 20:19:51.064145 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/master-0/wal/instance
I20260928 20:19:51.064273 9305 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:19:51.064391 3144 external_mini_cluster.cc:1428] Setting key 47e6904c1984e72945e247d0cdae7212
I20260928 20:19:51.064940 9305 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:19:51.066668 9305 catalog_manager.cc:1347] Generated new cluster ID: 44e10b6ab55043b4b4c53ad9882d8ab4
I20260928 20:19:51.066723 9305 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 20:19:51 dist-test-slave-b83v krb5kdc[9212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM
I20260928 20:19:51.077621 9315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:51.065248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49878 (local address 127.3.18.62:42379)
0928 20:19:51.065515 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:51.065523 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:51.065546 (+ 23us) server_negotiation.cc:407] Connection header received
0928 20:19:51.065642 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:51.065647 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:51.065724 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:51.065864 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:51.066670 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:51.068104 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:51.069219 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:51.069890 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:51.070009 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:51.073358 (+ 3349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:51.073364 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:51.073365 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:51.073381 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:51.075688 (+ 2307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:51.076186 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:51.076189 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:51.076190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:51.076323 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:51.076554 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:51.076555 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:51.076555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:51.076902 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:51.077070 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:51.077174 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 20:19:51.077326 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":81,"threads_started":1}
I20260928 20:19:51.088812 9305 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:19:51.089496 9305 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:19:51.099455 9305 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef4554cccf754a53a7d74f685421731a: Generated new TSK 0
I20260928 20:19:51.099709 9305 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.3.18.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.18.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.18.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/krb5kdc/oryx_127.3.18.1.keytab.
Entry for principal oryx/127.3.18.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/krb5kdc/oryx_127.3.18.1.keytab.
I20260928 20:19:51.136447 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.1:0
--local_ip_for_outbound_sockets=127.3.18.1
--webserver_interface=127.3.18.1
--webserver_port=0
--tserver_master_addrs=127.3.18.62:42379
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/krb5kdc/oryx_127.3.18.1.keytab
--principal=oryx/127.3.18.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:45321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:19:51.223857 9319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:19:51.224098 9319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:19:51.224128 9319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:19:51.225337 9319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:19:51.225379 9319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:19:51.225406 9319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.1
I20260928 20:19:51.226840 9319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:45321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/krb5kdc/oryx_127.3.18.1.keytab
--principal=oryx/127.3.18.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.18.1
--webserver_port=0
--tserver_master_addrs=127.3.18.62:42379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.1
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:19:51.227109 9319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:19:51.227325 9319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:19:51.233067 9325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:51.233071 9327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:51.233071 9324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:51.233094 9319 server_base.cc:1034] running on GCE node
I20260928 20:19:51.233517 9319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:19:51.233771 9319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:19:51.234941 9319 hybrid_clock.cc:648] HybridClock initialized: now 1790626791234923 us; error 32 us; skew 500 ppm
Sep 28 20:19:51 dist-test-slave-b83v krb5kdc[9212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626791, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.18.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:19:51.238186 9319 init.cc:377] Logged in from keytab as oryx/127.3.18.1@KRBTEST.COM (short username oryx)
I20260928 20:19:51.238821 9319 webserver.cc:460] Webserver started at http://127.3.18.1:42685/ using document root <none> and password file <none>
I20260928 20:19:51.239109 9319 fs_manager.cc:359] Metadata directory not provided
I20260928 20:19:51.239190 9319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:19:51.239360 9319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:19:51.240617 9319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-0/data/instance:
uuid: "39e67e2773bb4225a13ec3d8425a088a"
format_stamp: "Formatted at 2026-09-28 20:19:51 on dist-test-slave-b83v"
server_key: "250c489c34064eb62142fc0050022ed4"
server_key_iv: "54d155614f6cf7c60a419ac85a843a3b"
server_key_version: "encryptionkey@0"
I20260928 20:19:51.241045 9319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-0/wal/instance:
uuid: "39e67e2773bb4225a13ec3d8425a088a"
format_stamp: "Formatted at 2026-09-28 20:19:51 on dist-test-slave-b83v"
server_key: "250c489c34064eb62142fc0050022ed4"
server_key_iv: "54d155614f6cf7c60a419ac85a843a3b"
server_key_version: "encryptionkey@0"
I20260928 20:19:51.242759 9319 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 20:19:51.243881 9334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:51.244107 9319 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:19:51.244232 9319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-0/wal
uuid: "39e67e2773bb4225a13ec3d8425a088a"
format_stamp: "Formatted at 2026-09-28 20:19:51 on dist-test-slave-b83v"
server_key: "250c489c34064eb62142fc0050022ed4"
server_key_iv: "54d155614f6cf7c60a419ac85a843a3b"
server_key_version: "encryptionkey@0"
I20260928 20:19:51.244328 9319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:19:51.266209 9319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:19:51.268953 9319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:19:51.269160 9319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:19:51.269465 9319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:19:51.269848 9319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:19:51.269924 9319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:51.269971 9319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:19:51.270004 9319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:51.276983 9319 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.1:44945
I20260928 20:19:51.277042 9447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.1:44945 every 8 connection(s)
I20260928 20:19:51.277431 9319 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
I20260928 20:19:51.282650 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 9319
I20260928 20:19:51.282778 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-0/wal/instance
I20260928 20:19:51.283059 3144 external_mini_cluster.cc:1428] Setting key 0f2662b61e2c649c0b68d62a7a2804fe
Sep 28 20:19:51 dist-test-slave-b83v krb5kdc[9212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626791, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.18.1@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM
I20260928 20:19:51.289114 9315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:51.278320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:45115 (local address 127.3.18.62:42379)
0928 20:19:51.278385 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:51.278388 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:51.278646 (+ 258us) server_negotiation.cc:407] Connection header received
0928 20:19:51.279225 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:51.279226 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:51.279238 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:51.279300 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:51.280423 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:51.280972 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:51.282219 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:51.282438 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:51.282549 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:51.285536 (+ 2987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:51.285538 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:51.285539 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:51.285550 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:51.287371 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:51.287895 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:51.287896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:51.287897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:51.287964 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:51.288301 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:51.288302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:51.288303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:51.288473 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:51.288548 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:51.288854 (+ 306us) server_negotiation.cc:299] Negotiation successful
0928 20:19:51.288969 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 20:19:51.289119 9450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:51.278307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:42379 (local address 127.3.18.1:45115)
0928 20:19:51.278506 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:51.278546 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:51.279120 (+ 574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:51.279386 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:51.279391 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:51.280027 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:51.280275 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:51.280281 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:51.281109 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:51.281110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:51.282104 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:51.282107 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:51.282601 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:51.282602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:51.282786 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:51.283502 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:19:51.283512 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:51.285382 (+ 1870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:51.287461 (+ 2079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:51.287468 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:51.287470 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:51.287730 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:51.288053 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:51.288055 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:51.288055 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:51.288211 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:51.288544 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:51.288549 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:51.288691 (+ 142us) client_negotiation.cc:769] Sending connection context
0928 20:19:51.288780 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 20:19:51.288927 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":97,"threads_started":1}
I20260928 20:19:51.289726 9448 heartbeater.cc:347] Connected to a master server at 127.3.18.62:42379
I20260928 20:19:51.289839 9448 heartbeater.cc:464] Registering TS with master...
I20260928 20:19:51.290040 9448 heartbeater.cc:511] Master 127.3.18.62:42379 requested a full tablet report, sending...
I20260928 20:19:51.290685 9257 ts_manager.cc:194] Registered new tserver with Master: 39e67e2773bb4225a13ec3d8425a088a (127.3.18.1:44945)
I20260928 20:19:51.291374 9257 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.18.1@KRBTEST.COM'} at 127.3.18.1:45115
WARNING: no policy specified for oryx/127.3.18.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.18.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.18.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/krb5kdc/oryx_127.3.18.2.keytab.
Entry for principal oryx/127.3.18.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/krb5kdc/oryx_127.3.18.2.keytab.
I20260928 20:19:51.315706 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.2:0
--local_ip_for_outbound_sockets=127.3.18.2
--webserver_interface=127.3.18.2
--webserver_port=0
--tserver_master_addrs=127.3.18.62:42379
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/krb5kdc/oryx_127.3.18.2.keytab
--principal=oryx/127.3.18.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:45321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:19:51.401969 9453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:19:51.402254 9453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:19:51.402302 9453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:19:51.403517 9453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:19:51.403589 9453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:19:51.403631 9453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.2
I20260928 20:19:51.405171 9453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:45321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/krb5kdc/oryx_127.3.18.2.keytab
--principal=oryx/127.3.18.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.18.2
--webserver_port=0
--tserver_master_addrs=127.3.18.62:42379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.2
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:19:51.405469 9453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:19:51.405726 9453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:19:51.411316 9458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:51.411384 9461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:51.411319 9459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:51.411722 9453 server_base.cc:1034] running on GCE node
I20260928 20:19:51.412020 9453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:19:51.412305 9453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:19:51.413544 9453 hybrid_clock.cc:648] HybridClock initialized: now 1790626791413534 us; error 35 us; skew 500 ppm
Sep 28 20:19:51 dist-test-slave-b83v krb5kdc[9212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626791, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.18.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:19:51.416425 9453 init.cc:377] Logged in from keytab as oryx/127.3.18.2@KRBTEST.COM (short username oryx)
I20260928 20:19:51.416942 9453 webserver.cc:460] Webserver started at http://127.3.18.2:37573/ using document root <none> and password file <none>
I20260928 20:19:51.417196 9453 fs_manager.cc:359] Metadata directory not provided
I20260928 20:19:51.417243 9453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:19:51.417368 9453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:19:51.418455 9453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-1/data/instance:
uuid: "14a5194f37a543ad9cf9c8130a5d11af"
format_stamp: "Formatted at 2026-09-28 20:19:51 on dist-test-slave-b83v"
server_key: "244c25b2bda5e079d7739d4cf66d0c5e"
server_key_iv: "d770da6922462fb5921b98e0d6e1a9a5"
server_key_version: "encryptionkey@0"
I20260928 20:19:51.418808 9453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-1/wal/instance:
uuid: "14a5194f37a543ad9cf9c8130a5d11af"
format_stamp: "Formatted at 2026-09-28 20:19:51 on dist-test-slave-b83v"
server_key: "244c25b2bda5e079d7739d4cf66d0c5e"
server_key_iv: "d770da6922462fb5921b98e0d6e1a9a5"
server_key_version: "encryptionkey@0"
I20260928 20:19:51.420238 9453 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:19:51.421340 9468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:51.421669 9453 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:19:51.421780 9453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-1/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-1/wal
uuid: "14a5194f37a543ad9cf9c8130a5d11af"
format_stamp: "Formatted at 2026-09-28 20:19:51 on dist-test-slave-b83v"
server_key: "244c25b2bda5e079d7739d4cf66d0c5e"
server_key_iv: "d770da6922462fb5921b98e0d6e1a9a5"
server_key_version: "encryptionkey@0"
I20260928 20:19:51.421891 9453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:19:51.444085 9453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:19:51.446551 9453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:19:51.446723 9453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:19:51.447041 9453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:19:51.447474 9453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:19:51.447538 9453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:51.447578 9453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:19:51.447602 9453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:51.454190 9453 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.2:35465
I20260928 20:19:51.454231 9581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.2:35465 every 8 connection(s)
I20260928 20:19:51.454594 9453 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
Sep 28 20:19:51 dist-test-slave-b83v krb5kdc[9212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626791, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.18.2@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM
I20260928 20:19:51.461668 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 9453
I20260928 20:19:51.461797 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-1/wal/instance
I20260928 20:19:51.462018 3144 external_mini_cluster.cc:1428] Setting key 0e660f98978fca53fd59b766dc472674
I20260928 20:19:51.465704 9315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:51.455514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:49131 (local address 127.3.18.62:42379)
0928 20:19:51.455587 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:51.455590 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:51.455860 (+ 270us) server_negotiation.cc:407] Connection header received
0928 20:19:51.456404 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:51.456406 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:51.456422 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:51.456502 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:51.457642 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:51.458162 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:51.459278 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:51.459496 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:51.459559 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:51.462093 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:51.462095 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:51.462096 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:51.462107 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:51.464014 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:51.464503 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:51.464505 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:51.464505 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:51.464593 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:51.464907 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:51.464908 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:51.464909 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:51.465093 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:51.465176 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:51.465462 (+ 286us) server_negotiation.cc:299] Negotiation successful
0928 20:19:51.465562 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 20:19:51.465798 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:51.455446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:42379 (local address 127.3.18.2:49131)
0928 20:19:51.455667 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:51.455716 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:51.456273 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:51.456612 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:51.456617 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:51.457297 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:51.457525 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:51.457531 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:51.458301 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:51.458302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:51.459179 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:51.459182 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:51.459580 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:51.459581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:51.459692 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:51.460370 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:19:51.460379 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:51.461987 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:51.464111 (+ 2124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:51.464119 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:51.464122 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:51.464386 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:51.464689 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:51.464691 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:51.464691 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:51.464837 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:51.465173 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:51.465179 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:51.465319 (+ 140us) client_negotiation.cc:769] Sending connection context
0928 20:19:51.465411 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 20:19:51.465580 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":119,"threads_started":1}
I20260928 20:19:51.466322 9582 heartbeater.cc:347] Connected to a master server at 127.3.18.62:42379
I20260928 20:19:51.466444 9582 heartbeater.cc:464] Registering TS with master...
I20260928 20:19:51.466636 9582 heartbeater.cc:511] Master 127.3.18.62:42379 requested a full tablet report, sending...
I20260928 20:19:51.467124 9257 ts_manager.cc:194] Registered new tserver with Master: 14a5194f37a543ad9cf9c8130a5d11af (127.3.18.2:35465)
I20260928 20:19:51.467523 9257 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.18.2@KRBTEST.COM'} at 127.3.18.2:49131
WARNING: no policy specified for oryx/127.3.18.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.18.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.18.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/krb5kdc/oryx_127.3.18.3.keytab.
Entry for principal oryx/127.3.18.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/krb5kdc/oryx_127.3.18.3.keytab.
I20260928 20:19:51.492705 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.3:0
--local_ip_for_outbound_sockets=127.3.18.3
--webserver_interface=127.3.18.3
--webserver_port=0
--tserver_master_addrs=127.3.18.62:42379
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/krb5kdc/oryx_127.3.18.3.keytab
--principal=oryx/127.3.18.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:45321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:19:51.581342 9587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:19:51.581560 9587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:19:51.581584 9587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:19:51.582785 9587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:19:51.582830 9587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:19:51.582857 9587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.3
I20260928 20:19:51.584369 9587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:45321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/krb5kdc/oryx_127.3.18.3.keytab
--principal=oryx/127.3.18.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.18.3
--webserver_port=0
--tserver_master_addrs=127.3.18.62:42379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.3
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:19:51.584633 9587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:19:51.584872 9587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:19:51.590490 9592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:51.590606 9593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:51.590577 9587 server_base.cc:1034] running on GCE node
W20260928 20:19:51.590617 9595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:51.591004 9587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:19:51.591281 9587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:19:51.592497 9587 hybrid_clock.cc:648] HybridClock initialized: now 1790626791592443 us; error 35 us; skew 500 ppm
Sep 28 20:19:51 dist-test-slave-b83v krb5kdc[9212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626791, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.18.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:19:51.595681 9587 init.cc:377] Logged in from keytab as oryx/127.3.18.3@KRBTEST.COM (short username oryx)
I20260928 20:19:51.596294 9587 webserver.cc:460] Webserver started at http://127.3.18.3:44589/ using document root <none> and password file <none>
I20260928 20:19:51.596566 9587 fs_manager.cc:359] Metadata directory not provided
I20260928 20:19:51.596626 9587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:19:51.596764 9587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:19:51.597887 9587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-2/data/instance:
uuid: "1f0e35e0a8fd4cec9d631c0701e0b76f"
format_stamp: "Formatted at 2026-09-28 20:19:51 on dist-test-slave-b83v"
server_key: "f5c1123f66a95cd87d7d92578aac3bb0"
server_key_iv: "9b650ad1c1697cecb65b366777c4c0b0"
server_key_version: "encryptionkey@0"
I20260928 20:19:51.598265 9587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-2/wal/instance:
uuid: "1f0e35e0a8fd4cec9d631c0701e0b76f"
format_stamp: "Formatted at 2026-09-28 20:19:51 on dist-test-slave-b83v"
server_key: "f5c1123f66a95cd87d7d92578aac3bb0"
server_key_iv: "9b650ad1c1697cecb65b366777c4c0b0"
server_key_version: "encryptionkey@0"
I20260928 20:19:51.599735 9587 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:19:51.600745 9602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:51.600979 9587 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:19:51.601086 9587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-2/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-2/wal
uuid: "1f0e35e0a8fd4cec9d631c0701e0b76f"
format_stamp: "Formatted at 2026-09-28 20:19:51 on dist-test-slave-b83v"
server_key: "f5c1123f66a95cd87d7d92578aac3bb0"
server_key_iv: "9b650ad1c1697cecb65b366777c4c0b0"
server_key_version: "encryptionkey@0"
I20260928 20:19:51.601174 9587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:19:51.623713 9587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:19:51.626147 9587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:19:51.626320 9587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:19:51.626613 9587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:19:51.627050 9587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:19:51.627094 9587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:51.627120 9587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:19:51.627131 9587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:51.633751 9587 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.3:38189
I20260928 20:19:51.633828 9715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.3:38189 every 8 connection(s)
I20260928 20:19:51.634198 9587 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
I20260928 20:19:51.638706 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 9587
I20260928 20:19:51.638801 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-2/wal/instance
I20260928 20:19:51.639010 3144 external_mini_cluster.cc:1428] Setting key dfeb38154c8376f25757b87da086119a
Sep 28 20:19:51 dist-test-slave-b83v krb5kdc[9212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626791, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.18.3@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM
I20260928 20:19:51.645323 9315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:51.635012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:39213 (local address 127.3.18.62:42379)
0928 20:19:51.635093 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:51.635095 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:51.635366 (+ 271us) server_negotiation.cc:407] Connection header received
0928 20:19:51.636028 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:51.636029 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:51.636040 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:51.636100 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:51.637192 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:51.637739 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:51.639151 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:51.639466 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:51.639535 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:51.642093 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:51.642095 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:51.642095 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:51.642104 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:51.643688 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:51.644201 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:51.644202 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:51.644203 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:51.644269 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:51.644628 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:51.644629 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:51.644629 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:51.644758 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:51.644817 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:51.645172 (+ 355us) server_negotiation.cc:299] Negotiation successful
0928 20:19:51.645253 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 20:19:51.645607 9718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:51.634985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:42379 (local address 127.3.18.3:39213)
0928 20:19:51.635192 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:51.635257 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:51.635901 (+ 644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:51.636178 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:51.636182 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:51.636799 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:51.637041 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:51.637047 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:51.637873 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:51.637874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:51.639013 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:51.639016 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:51.639559 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:51.639560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:51.639740 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:51.640388 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:19:51.640399 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:51.641978 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:51.643837 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:51.643853 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:51.643856 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:51.644123 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:51.644385 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:51.644385 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:51.644386 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:51.644520 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:51.644853 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:51.644860 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:51.645016 (+ 156us) client_negotiation.cc:769] Sending connection context
0928 20:19:51.645116 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 20:19:51.645265 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":103,"threads_started":1}
I20260928 20:19:51.646209 9716 heartbeater.cc:347] Connected to a master server at 127.3.18.62:42379
I20260928 20:19:51.646361 9716 heartbeater.cc:464] Registering TS with master...
I20260928 20:19:51.646572 9716 heartbeater.cc:511] Master 127.3.18.62:42379 requested a full tablet report, sending...
I20260928 20:19:51.647046 9257 ts_manager.cc:194] Registered new tserver with Master: 1f0e35e0a8fd4cec9d631c0701e0b76f (127.3.18.3:38189)
I20260928 20:19:51.647467 9257 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.18.3@KRBTEST.COM'} at 127.3.18.3:39213
I20260928 20:19:51.651188 3144 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.3.18.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.18.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.18.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/krb5kdc/oryx_127.3.18.61.keytab.
Entry for principal oryx/127.3.18.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/krb5kdc/oryx_127.3.18.61.keytab.
I20260928 20:19:51.680969 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.61:33799
--webserver_interface=127.3.18.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/krb5kdc/oryx_127.3.18.61.keytab
--principal=oryx/127.3.18.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:45321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.18.62:42379,127.3.18.61:33799
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:19:51.769804 9721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:19:51.770073 9721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:19:51.770116 9721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:19:51.771353 9721 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:19:51.771425 9721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:19:51.771447 9721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:19:51.771467 9721 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:19:51.771497 9721 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:19:51.773052 9721 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:45321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.18.62:42379,127.3.18.61:33799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/krb5kdc/oryx_127.3.18.61.keytab
--principal=oryx/127.3.18.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.61:33799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.18.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
W20260928 20:19:51.773211 9721 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.3.18.62:42379,127.3.18.61:33799'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 20:19:51.773375 9721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:19:51.773600 9721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:19:51.778728 9727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:51.778736 9726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:51.778769 9729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:51.778949 9721 server_base.cc:1034] running on GCE node
I20260928 20:19:51.779242 9721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:19:51.779563 9721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:19:51.780756 9721 hybrid_clock.cc:648] HybridClock initialized: now 1790626791780733 us; error 55 us; skew 500 ppm
Sep 28 20:19:51 dist-test-slave-b83v krb5kdc[9212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626791, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.18.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:19:51.783501 9721 init.cc:377] Logged in from keytab as oryx/127.3.18.61@KRBTEST.COM (short username oryx)
I20260928 20:19:51.784065 9721 webserver.cc:460] Webserver started at http://127.3.18.61:42783/ using document root <none> and password file <none>
I20260928 20:19:51.784317 9721 fs_manager.cc:359] Metadata directory not provided
I20260928 20:19:51.784386 9721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:19:51.784536 9721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:19:51.785663 9721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/master-1/data/instance:
uuid: "d13b62b954784d05bd09150e14f708b7"
format_stamp: "Formatted at 2026-09-28 20:19:51 on dist-test-slave-b83v"
server_key: "f2240220bb563b0f49dd834993f54634"
server_key_iv: "1a72ad6c8243dd932afa49a2a9cd04d6"
server_key_version: "encryptionkey@0"
I20260928 20:19:51.786036 9721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/master-1/wal/instance:
uuid: "d13b62b954784d05bd09150e14f708b7"
format_stamp: "Formatted at 2026-09-28 20:19:51 on dist-test-slave-b83v"
server_key: "f2240220bb563b0f49dd834993f54634"
server_key_iv: "1a72ad6c8243dd932afa49a2a9cd04d6"
server_key_version: "encryptionkey@0"
I20260928 20:19:51.787497 9721 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:19:51.788420 9736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:51.788599 9721 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 20:19:51.788707 9721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/master-1/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/master-1/wal
uuid: "d13b62b954784d05bd09150e14f708b7"
format_stamp: "Formatted at 2026-09-28 20:19:51 on dist-test-slave-b83v"
server_key: "f2240220bb563b0f49dd834993f54634"
server_key_iv: "1a72ad6c8243dd932afa49a2a9cd04d6"
server_key_version: "encryptionkey@0"
I20260928 20:19:51.788784 9721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:19:51.817238 9721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:19:51.819715 9721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:19:51.819911 9721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:19:51.824195 9721 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.61:33799
I20260928 20:19:51.824265 9788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.61:33799 every 8 connection(s)
I20260928 20:19:51.824637 9721 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/master-1/data/info.pb
I20260928 20:19:51.825726 9789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:51.826579 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 9721
I20260928 20:19:51.827885 9789 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.18.62" port: 42379 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 20:19:51 dist-test-slave-b83v krb5kdc[9212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626791, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.18.61@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM
Sep 28 20:19:51 dist-test-slave-b83v krb5kdc[9212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.18.61@KRBTEST.COM
I20260928 20:19:51.837724 9315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:51.828655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49884 (local address 127.3.18.62:42379)
0928 20:19:51.828712 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:51.828714 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:51.828813 (+ 99us) server_negotiation.cc:407] Connection header received
0928 20:19:51.828865 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:51.828867 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:51.828878 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:51.828945 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:51.829528 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:51.830026 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:51.830976 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:51.831260 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:51.831334 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:51.833887 (+ 2553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:51.833889 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:51.833889 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:51.833898 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:51.836141 (+ 2243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:51.836536 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:51.836538 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:51.836538 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:51.836602 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:51.836884 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:51.836885 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:51.836885 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:51.837171 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:51.837245 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:51.837543 (+ 298us) server_negotiation.cc:299] Negotiation successful
0928 20:19:51.837628 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 20:19:51.837953 9793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:51.828509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:42379 (local address 127.0.0.1:49884)
0928 20:19:51.828709 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:51.828747 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:51.828818 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:51.828971 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:51.828974 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:51.829301 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:51.829410 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:51.829416 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:51.830153 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:51.830153 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:51.830894 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:51.830897 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:51.831342 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:51.831343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:51.831413 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:51.832171 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:19:51.832181 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:51.833790 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:51.836234 (+ 2444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:51.836239 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:51.836241 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:51.836450 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:51.836673 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:51.836674 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:51.836675 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:51.836817 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:51.837248 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:51.837255 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:51.837406 (+ 151us) client_negotiation.cc:769] Sending connection context
0928 20:19:51.837520 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 20:19:51.837683 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":124,"threads_started":1}
I20260928 20:19:51.838644 9792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:51.827280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50122 (local address 127.3.18.61:33799)
0928 20:19:51.827448 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:51.827458 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:51.827484 (+ 26us) server_negotiation.cc:407] Connection header received
0928 20:19:51.827963 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:51.827970 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:51.828020 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:51.828136 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:51.828934 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:51.830009 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:51.830786 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:51.831211 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:51.831326 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:51.834640 (+ 3314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:51.834648 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:51.834649 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:51.834656 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:51.836908 (+ 2252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:51.837266 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:51.837270 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:51.837270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:51.837323 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:51.837544 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:51.837545 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:51.837545 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:51.837734 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:51.837925 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:51.838449 (+ 524us) server_negotiation.cc:299] Negotiation successful
0928 20:19:51.838542 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":81,"threads_started":1}
I20260928 20:19:51.838625 9789 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.18.61" port: 33799 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 20:19:51 dist-test-slave-b83v krb5kdc[9212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626791, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.18.61@KRBTEST.COM for oryx/127.3.18.61@KRBTEST.COM
I20260928 20:19:51.845288 9793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:51.839055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.61:33799 (local address 127.0.0.1:50138)
0928 20:19:51.839092 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:51.839100 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:51.839134 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:51.839370 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:51.839371 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:51.839593 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:51.839639 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:51.839641 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:51.840326 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:51.840327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:51.840812 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:51.840814 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:51.841134 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:51.841135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:51.841182 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:51.841540 (+ 358us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:19:51.841544 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:51.842674 (+ 1130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:51.844253 (+ 1579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:51.844254 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:51.844254 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:51.844468 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:51.844689 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:51.844690 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:51.844690 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:51.844767 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:51.845044 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:51.845045 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:51.845080 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 20:19:51.845143 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 20:19:51.845220 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260928 20:19:51.845412 9792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:51.839156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50138 (local address 127.3.18.61:33799)
0928 20:19:51.839200 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:51.839202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:51.839210 (+ 8us) server_negotiation.cc:407] Connection header received
0928 20:19:51.839229 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:51.839230 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:51.839250 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:51.839322 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:51.839727 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:51.840254 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:51.840881 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:51.841073 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:51.841124 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:51.842757 (+ 1633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:51.842758 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:51.842758 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:51.842764 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:51.844173 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:51.844562 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:51.844563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:51.844563 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:51.844615 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:51.844854 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:51.844855 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:51.844855 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:51.844975 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:51.845039 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:51.845160 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 20:19:51.845266 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":26}
I20260928 20:19:51.847105 9789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d13b62b954784d05bd09150e14f708b7: Bootstrap starting.
I20260928 20:19:51.848228 9789 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d13b62b954784d05bd09150e14f708b7: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:51.848557 9789 log.cc:824] T 00000000000000000000000000000000 P d13b62b954784d05bd09150e14f708b7: Log is configured to *not* fsync() on all Append() calls
I20260928 20:19:51.849447 9789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d13b62b954784d05bd09150e14f708b7: No bootstrap required, opened a new log
I20260928 20:19:51.851008 9789 raft_consensus.cc:348] T 00000000000000000000000000000000 P d13b62b954784d05bd09150e14f708b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef4554cccf754a53a7d74f685421731a" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 42379 } } peers { permanent_uuid: "d13b62b954784d05bd09150e14f708b7" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 33799 } }
I20260928 20:19:51.851162 9789 raft_consensus.cc:374] T 00000000000000000000000000000000 P d13b62b954784d05bd09150e14f708b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:51.851187 9789 raft_consensus.cc:729] T 00000000000000000000000000000000 P d13b62b954784d05bd09150e14f708b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13b62b954784d05bd09150e14f708b7, State: Initialized, Role: FOLLOWER
I20260928 20:19:51.851328 9789 consensus_queue.cc:260] T 00000000000000000000000000000000 P d13b62b954784d05bd09150e14f708b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef4554cccf754a53a7d74f685421731a" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 42379 } } peers { permanent_uuid: "d13b62b954784d05bd09150e14f708b7" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 33799 } }
I20260928 20:19:51.851686 9789 sys_catalog.cc:564] T 00000000000000000000000000000000 P d13b62b954784d05bd09150e14f708b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:19:51.851670 9795 sys_catalog.cc:455] T 00000000000000000000000000000000 P d13b62b954784d05bd09150e14f708b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef4554cccf754a53a7d74f685421731a" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 42379 } } peers { permanent_uuid: "d13b62b954784d05bd09150e14f708b7" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 33799 } } }
I20260928 20:19:51.851840 9795 sys_catalog.cc:458] T 00000000000000000000000000000000 P d13b62b954784d05bd09150e14f708b7 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 20:19:51.853713 9806 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d13b62b954784d05bd09150e14f708b7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 20:19:51.853811 9806 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 20:19:51.855491 9721 master_runner.cc:428] Detected that this master d13b62b954784d05bd09150e14f708b7 is joining an existing cluster
I20260928 20:19:51.855556 9721 master_runner.cc:432] Initiating AddMaster RPC to add 127.3.18.61:33799
I20260928 20:19:51.864344 9315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:51.856834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49886 (local address 127.3.18.62:42379)
0928 20:19:51.856897 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:51.856899 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:51.857044 (+ 145us) server_negotiation.cc:407] Connection header received
0928 20:19:51.857133 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:51.857134 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:51.857145 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:51.857212 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:51.857732 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:51.858485 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:51.859493 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:51.859828 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:51.859903 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:51.861250 (+ 1347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:51.861251 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:51.861252 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:51.861260 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:51.862871 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:51.863347 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:51.863348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:51.863348 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:51.863475 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:51.863688 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:51.863689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:51.863689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:51.863866 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:51.863941 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:51.864148 (+ 207us) server_negotiation.cc:299] Negotiation successful
0928 20:19:51.864245 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 20:19:51.864337 9792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:51.856998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50150 (local address 127.3.18.61:33799)
0928 20:19:51.857060 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:51.857061 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:51.857221 (+ 160us) server_negotiation.cc:407] Connection header received
0928 20:19:51.857285 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:51.857286 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:51.857297 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:51.857338 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:51.857746 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:51.858201 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:51.859122 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:51.859401 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:51.859450 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:51.860706 (+ 1256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:51.860708 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:51.860708 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:51.860715 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:51.862819 (+ 2104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:51.863371 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:51.863373 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:51.863373 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:51.863450 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:51.863717 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:51.863719 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:51.863719 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:51.863866 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:51.863947 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:51.864130 (+ 183us) server_negotiation.cc:299] Negotiation successful
0928 20:19:51.864224 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 20:19:51.864749 9813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:51.856748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:42379 (local address 127.0.0.1:49886)
0928 20:19:51.856975 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:51.856987 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:51.857071 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:51.857253 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:51.857253 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:51.857522 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:51.857641 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:51.857644 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:51.858569 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:51.858570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:51.859406 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:51.859408 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:51.859929 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:51.859930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:51.859998 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:51.860770 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:19:51.860774 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:51.861123 (+ 349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:51.862950 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:51.862951 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:51.862951 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:51.863269 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:51.863538 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:51.863538 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:51.863539 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:51.863620 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:51.863969 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:51.863970 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:51.864022 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 20:19:51.864098 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 20:19:51.864279 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":35,"thread_start_us":92,"threads_started":1}
I20260928 20:19:51.864848 9812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:51.856758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.61:33799 (local address 127.0.0.1:50150)
0928 20:19:51.857153 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:51.857160 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:51.857230 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:51.857363 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:51.857363 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:51.857615 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:51.857666 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:51.857668 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:51.858289 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:51.858290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:51.859040 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:51.859042 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:51.859518 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:51.859519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:51.859568 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:51.860185 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:19:51.860190 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:51.860589 (+ 399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:51.862910 (+ 2321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:51.862911 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:51.862912 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:51.863284 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:51.863539 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:51.863540 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:51.863540 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:51.863643 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:51.863969 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:51.863970 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:51.864022 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 20:19:51.864442 (+ 420us) client_negotiation.cc:240] Negotiation successful
0928 20:19:51.864588 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":4,"thread_start_us":96,"threads_started":1}
Sep 28 20:19:51 dist-test-slave-b83v krb5kdc[9212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626791, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.18.62@KRBTEST.COM for oryx/127.3.18.61@KRBTEST.COM
I20260928 20:19:51.876006 9792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:51.867791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50154 (local address 127.3.18.61:33799)
0928 20:19:51.867838 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:51.867841 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:51.868065 (+ 224us) server_negotiation.cc:407] Connection header received
0928 20:19:51.868173 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:51.868174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:51.868187 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:51.868266 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:51.868791 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:51.869301 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:51.870192 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:51.870404 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:51.870502 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:51.872950 (+ 2448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:51.872951 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:51.872952 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:51.872958 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:51.874547 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:51.875077 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:51.875078 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:51.875079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:51.875146 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:51.875362 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:51.875363 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:51.875363 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:51.875495 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:51.875572 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:51.875808 (+ 236us) server_negotiation.cc:299] Negotiation successful
0928 20:19:51.875894 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 20:19:51.876017 9814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:51.867712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.61:33799 (local address 127.0.0.1:50154)
0928 20:19:51.867954 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:51.867996 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:51.868094 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:51.868274 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:51.868281 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:51.868627 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:51.868687 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:51.868689 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:51.869399 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:51.869401 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:51.870080 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:51.870082 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:51.870505 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:51.870506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:51.870611 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:51.871146 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:19:51.871151 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:51.872814 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:51.874714 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:51.874722 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:51.874725 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:51.874990 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:51.875220 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:51.875221 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:51.875222 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:51.875292 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:51.875575 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:51.875583 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:51.875650 (+ 67us) client_negotiation.cc:769] Sending connection context
0928 20:19:51.875744 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 20:19:51.875917 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":108,"threads_started":1}
I20260928 20:19:51.876609 9257 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.3.18.61:33799: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "d13b62b954784d05bd09150e14f708b7"
member_type: NON_VOTER
last_known_addr {
host: "127.3.18.61"
port: 33799
}
attrs {
promote: true
}
}
dest_uuid: "ef4554cccf754a53a7d74f685421731a"
cas_config_opid_index: -1
I20260928 20:19:51.876972 9257 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef4554cccf754a53a7d74f685421731a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ef4554cccf754a53a7d74f685421731a" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 42379 } } peers { permanent_uuid: "d13b62b954784d05bd09150e14f708b7" member_type: NON_VOTER last_known_addr { host: "127.3.18.61" port: 33799 } attrs { promote: true } }
I20260928 20:19:51.877498 9816 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ef4554cccf754a53a7d74f685421731a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER d13b62b954784d05bd09150e14f708b7 (127.3.18.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ef4554cccf754a53a7d74f685421731a" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 42379 } } peers { permanent_uuid: "d13b62b954784d05bd09150e14f708b7" member_type: NON_VOTER last_known_addr { host: "127.3.18.61" port: 33799 } attrs { promote: true } } }
I20260928 20:19:51.877969 9763 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d13b62b954784d05bd09150e14f708b7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:51.878976 9816 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.3.18.61:33799
I20260928 20:19:51.879074 9816 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef4554cccf754a53a7d74f685421731a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ef4554cccf754a53a7d74f685421731a" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ef4554cccf754a53a7d74f685421731a" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 42379 } } peers { permanent_uuid: "d13b62b954784d05bd09150e14f708b7" member_type: NON_VOTER last_known_addr { host: "127.3.18.61" port: 33799 } attrs { promote: true } } }
I20260928 20:19:51.879137 9816 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef4554cccf754a53a7d74f685421731a [sys.catalog]: This master's current role is: LEADER
I20260928 20:19:51.879127 9763 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d13b62b954784d05bd09150e14f708b7 [term 1 FOLLOWER]: Refusing update from remote peer ef4554cccf754a53a7d74f685421731a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 20:19:51.879259 9721 master.cc:537] Master@127.3.18.61:33799 shutting down...
I20260928 20:19:51.879447 9816 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ef4554cccf754a53a7d74f685421731a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13b62b954784d05bd09150e14f708b7" member_type: NON_VOTER last_known_addr { host: "127.3.18.61" port: 33799 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 20:19:51.880259 9241 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260928 20:19:51.880698 9241 consensus_peers.cc:487] T 00000000000000000000000000000000 P ef4554cccf754a53a7d74f685421731a -> Peer d13b62b954784d05bd09150e14f708b7 (127.3.18.61:33799): Couldn't send request to peer d13b62b954784d05bd09150e14f708b7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 20:19:51.882072 9721 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d13b62b954784d05bd09150e14f708b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:19:51.882169 9721 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d13b62b954784d05bd09150e14f708b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:19:51.882242 9721 tablet_replica.cc:330] T 00000000000000000000000000000000 P d13b62b954784d05bd09150e14f708b7: stopping tablet replica
I20260928 20:19:51.895454 9721 master.cc:552] Master@127.3.18.61:33799 shutdown complete.
I20260928 20:19:51.895514 9721 master_runner.cc:305] Clearing existing system tablet
I20260928 20:19:51.895687 9721 fs_manager.cc:359] Metadata directory not provided
I20260928 20:19:51.895738 9721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:19:51.896510 9721 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:51.897125 9820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:51.897270 9721 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:19:51.897343 9721 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 20:19:51.897392 9721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/master-1/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/master-1/wal
uuid: "d13b62b954784d05bd09150e14f708b7"
format_stamp: "Formatted at 2026-09-28 20:19:51 on dist-test-slave-b83v"
server_key: "f2240220bb563b0f49dd834993f54634"
server_key_iv: "1a72ad6c8243dd932afa49a2a9cd04d6"
server_key_version: "encryptionkey@0"
I20260928 20:19:51.898304 9721 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P d13b62b954784d05bd09150e14f708b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:19:51.900383 9721 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P d13b62b954784d05bd09150e14f708b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 20:19:51.900481 9721 log.cc:1196] T 00000000000000000000000000000000 P d13b62b954784d05bd09150e14f708b7: Deleting WAL directory at /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 20:19:51.900715 9721 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P d13b62b954784d05bd09150e14f708b7: Deleting consensus metadata
I20260928 20:19:51.900852 9721 master_runner.cc:315] Copying system tablet from 127.3.18.62:42379
I20260928 20:19:51.901602 9721 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P d13b62b954784d05bd09150e14f708b7: tablet copy: Beginning tablet copy session from remote peer at address 127.3.18.62:42379
I20260928 20:19:51.909054 9826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:51.901943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:42379 (local address 127.0.0.1:49902)
0928 20:19:51.902145 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:51.902159 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:51.902246 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:51.902459 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:51.902460 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:51.902692 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:51.902773 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:51.902775 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:51.903582 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:51.903584 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:51.904262 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:51.904264 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:51.904839 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:51.904841 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:51.904900 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:51.905464 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:19:51.905469 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:51.905830 (+ 361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:51.907985 (+ 2155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:51.907986 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:51.907987 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:51.908171 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:51.908415 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:51.908416 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:51.908416 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:51.908482 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:51.908770 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:51.908771 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:51.908812 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 20:19:51.908875 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 20:19:51.908965 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":136,"threads_started":1}
I20260928 20:19:51.909088 9315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:51.902070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49902 (local address 127.3.18.62:42379)
0928 20:19:51.902134 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:51.902142 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:51.902282 (+ 140us) server_negotiation.cc:407] Connection header received
0928 20:19:51.902315 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:51.902316 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:51.902350 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:51.902429 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:51.902864 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:51.903438 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:51.904351 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:51.904731 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:51.904814 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:51.905933 (+ 1119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:51.905935 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:51.905935 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:51.905944 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:51.907743 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:51.908243 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:51.908245 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:51.908245 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:51.908314 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:51.908557 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:51.908558 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:51.908558 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:51.908703 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:51.908774 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:51.908906 (+ 132us) server_negotiation.cc:299] Negotiation successful
0928 20:19:51.908990 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 20:19:51.909895 9277 tablet_copy_service.cc:140] P ef4554cccf754a53a7d74f685421731a: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer d13b62b954784d05bd09150e14f708b7 ({username='oryx', principal='oryx/127.3.18.61@KRBTEST.COM'} at 127.0.0.1:49902)
I20260928 20:19:51.910032 9277 tablet_copy_service.cc:161] P ef4554cccf754a53a7d74f685421731a: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer d13b62b954784d05bd09150e14f708b7 at {username='oryx', principal='oryx/127.3.18.61@KRBTEST.COM'} at 127.0.0.1:49902: session id = d13b62b954784d05bd09150e14f708b7-00000000000000000000000000000000
I20260928 20:19:51.910766 9277 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P ef4554cccf754a53a7d74f685421731a: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 20:19:51.911546 9721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 20:19:51.912997 9721 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P d13b62b954784d05bd09150e14f708b7: tablet copy: Starting download of 0 data blocks...
I20260928 20:19:51.913163 9721 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P d13b62b954784d05bd09150e14f708b7: tablet copy: Starting download of 1 WAL segments...
I20260928 20:19:51.914228 9721 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P d13b62b954784d05bd09150e14f708b7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 20:19:51.915174 9277 tablet_copy_service.cc:342] P ef4554cccf754a53a7d74f685421731a: Request end of tablet copy session d13b62b954784d05bd09150e14f708b7-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.3.18.61@KRBTEST.COM'} at 127.0.0.1:49902
I20260928 20:19:51.915261 9277 tablet_copy_service.cc:433] P ef4554cccf754a53a7d74f685421731a: ending tablet copy session d13b62b954784d05bd09150e14f708b7-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer d13b62b954784d05bd09150e14f708b7
I20260928 20:19:51.918275 9721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:19:51.919663 9827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:51.919821 9828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:51.919988 9830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:51.923955 9721 server_base.cc:1034] running on GCE node
I20260928 20:19:51.924098 9721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:19:51.924283 9721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:19:51.925410 9721 hybrid_clock.cc:648] HybridClock initialized: now 1790626791925401 us; error 32 us; skew 500 ppm
Sep 28 20:19:51 dist-test-slave-b83v krb5kdc[9212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626791, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.18.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:19:51.927062 9721 init.cc:377] Logged in from keytab as oryx/127.3.18.61@KRBTEST.COM (short username oryx)
I20260928 20:19:51.927403 9721 webserver.cc:460] Webserver started at http://127.3.18.61:37207/ using document root <none> and password file <none>
I20260928 20:19:51.927554 9721 fs_manager.cc:359] Metadata directory not provided
I20260928 20:19:51.927603 9721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:19:51.928370 9721 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:51.928993 9837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:51.929123 9721 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 20:19:51.929174 9721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/master-1/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/master-1/wal
uuid: "d13b62b954784d05bd09150e14f708b7"
format_stamp: "Formatted at 2026-09-28 20:19:51 on dist-test-slave-b83v"
server_key: "f2240220bb563b0f49dd834993f54634"
server_key_iv: "1a72ad6c8243dd932afa49a2a9cd04d6"
server_key_version: "encryptionkey@0"
I20260928 20:19:51.929236 9721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:19:51.971869 9721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:19:51.973769 9721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:19:51.977655 9721 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.61:33799
I20260928 20:19:51.977679 9889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.61:33799 every 8 connection(s)
I20260928 20:19:51.978044 9721 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/master-1/data/info.pb
I20260928 20:19:51.979194 9890 sys_catalog.cc:263] Verifying existing consensus state
I20260928 20:19:51.979660 9890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d13b62b954784d05bd09150e14f708b7: Bootstrap starting.
I20260928 20:19:51.982036 9890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d13b62b954784d05bd09150e14f708b7: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 20:19:51.982256 9890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d13b62b954784d05bd09150e14f708b7: Bootstrap complete.
I20260928 20:19:51.982460 9890 raft_consensus.cc:348] T 00000000000000000000000000000000 P d13b62b954784d05bd09150e14f708b7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ef4554cccf754a53a7d74f685421731a" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 42379 } } peers { permanent_uuid: "d13b62b954784d05bd09150e14f708b7" member_type: NON_VOTER last_known_addr { host: "127.3.18.61" port: 33799 } attrs { promote: true } }
I20260928 20:19:51.982549 9890 raft_consensus.cc:729] T 00000000000000000000000000000000 P d13b62b954784d05bd09150e14f708b7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d13b62b954784d05bd09150e14f708b7, State: Initialized, Role: LEARNER
I20260928 20:19:51.982600 9890 consensus_queue.cc:260] T 00000000000000000000000000000000 P d13b62b954784d05bd09150e14f708b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ef4554cccf754a53a7d74f685421731a" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 42379 } } peers { permanent_uuid: "d13b62b954784d05bd09150e14f708b7" member_type: NON_VOTER last_known_addr { host: "127.3.18.61" port: 33799 } attrs { promote: true } }
I20260928 20:19:51.982836 9890 sys_catalog.cc:564] T 00000000000000000000000000000000 P d13b62b954784d05bd09150e14f708b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:19:51.982815 9893 sys_catalog.cc:455] T 00000000000000000000000000000000 P d13b62b954784d05bd09150e14f708b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ef4554cccf754a53a7d74f685421731a" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 42379 } } peers { permanent_uuid: "d13b62b954784d05bd09150e14f708b7" member_type: NON_VOTER last_known_addr { host: "127.3.18.61" port: 33799 } attrs { promote: true } } }
I20260928 20:19:51.982906 9893 sys_catalog.cc:458] T 00000000000000000000000000000000 P d13b62b954784d05bd09150e14f708b7 [sys.catalog]: This master's current role is: LEARNER
I20260928 20:19:51.984298 9904 catalog_manager.cc:1259] Loaded cluster ID: 44e10b6ab55043b4b4c53ad9882d8ab4
I20260928 20:19:51.984375 9904 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d13b62b954784d05bd09150e14f708b7: loading cluster ID for follower catalog manager: success
I20260928 20:19:51.985569 9904 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d13b62b954784d05bd09150e14f708b7: acquiring CA information for follower catalog manager: success
I20260928 20:19:51.985898 9904 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d13b62b954784d05bd09150e14f708b7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 20:19:52.004374 9905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:51.991228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50192 (local address 127.3.18.61:33799)
0928 20:19:51.991385 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:51.991388 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:51.991396 (+ 8us) server_negotiation.cc:407] Connection header received
0928 20:19:51.991458 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:51.991459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:51.991471 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:51.991561 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:51.992195 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:51.992856 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:51.993647 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:51.993864 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:51.993974 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:52.001139 (+ 7165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:52.001142 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:52.001142 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:52.001159 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:52.002894 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:52.003397 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:52.003399 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:52.003400 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:52.003463 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:52.003740 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:52.003741 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:52.003742 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:52.003912 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:52.004052 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:52.004155 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 20:19:52.004247 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":80,"threads_started":1}
WARNING: no policy specified for oryx/127.3.18.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.18.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.18.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/krb5kdc/oryx_127.3.18.60.keytab.
Entry for principal oryx/127.3.18.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/krb5kdc/oryx_127.3.18.60.keytab.
I20260928 20:19:52.034878 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.60:38323
--webserver_interface=127.3.18.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/krb5kdc/oryx_127.3.18.60.keytab
--principal=oryx/127.3.18.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:45321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.18.62:42379,127.3.18.61:33799,127.3.18.60:38323
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:19:52.120158 9908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:19:52.120395 9908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:19:52.120430 9908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:19:52.121608 9908 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:19:52.121697 9908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:19:52.121722 9908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:19:52.121755 9908 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:19:52.121783 9908 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:19:52.123345 9908 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:45321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.18.62:42379,127.3.18.61:33799,127.3.18.60:38323
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/krb5kdc/oryx_127.3.18.60.keytab
--principal=oryx/127.3.18.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.60:38323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.18.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9908
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:19:52.123662 9908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:19:52.123953 9908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:19:52.129581 9914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:52.129638 9916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:52.129778 9908 server_base.cc:1034] running on GCE node
W20260928 20:19:52.129591 9913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:52.130157 9908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:19:52.130522 9908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:19:52.131690 9908 hybrid_clock.cc:648] HybridClock initialized: now 1790626792131680 us; error 33 us; skew 500 ppm
Sep 28 20:19:52 dist-test-slave-b83v krb5kdc[9212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626792, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.18.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:19:52.134616 9908 init.cc:377] Logged in from keytab as oryx/127.3.18.60@KRBTEST.COM (short username oryx)
I20260928 20:19:52.135134 9908 webserver.cc:460] Webserver started at http://127.3.18.60:33491/ using document root <none> and password file <none>
I20260928 20:19:52.135401 9908 fs_manager.cc:359] Metadata directory not provided
I20260928 20:19:52.135481 9908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:19:52.135641 9908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:19:52.136919 9908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/master-2/data/instance:
uuid: "645a6ba8e676428585dc367ef37372c4"
format_stamp: "Formatted at 2026-09-28 20:19:52 on dist-test-slave-b83v"
server_key: "973fb86d2949f05fc10c49e10ad8e6d8"
server_key_iv: "83e6f11a0d30707ce2c217739fc83d9b"
server_key_version: "encryptionkey@0"
I20260928 20:19:52.137319 9908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/master-2/wal/instance:
uuid: "645a6ba8e676428585dc367ef37372c4"
format_stamp: "Formatted at 2026-09-28 20:19:52 on dist-test-slave-b83v"
server_key: "973fb86d2949f05fc10c49e10ad8e6d8"
server_key_iv: "83e6f11a0d30707ce2c217739fc83d9b"
server_key_version: "encryptionkey@0"
I20260928 20:19:52.138823 9908 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 20:19:52.139861 9923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:52.140138 9908 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:19:52.140275 9908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/master-2/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/master-2/wal
uuid: "645a6ba8e676428585dc367ef37372c4"
format_stamp: "Formatted at 2026-09-28 20:19:52 on dist-test-slave-b83v"
server_key: "973fb86d2949f05fc10c49e10ad8e6d8"
server_key_iv: "83e6f11a0d30707ce2c217739fc83d9b"
server_key_version: "encryptionkey@0"
I20260928 20:19:52.140363 9908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:19:52.163632 9908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:19:52.166801 9908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:19:52.167201 9908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:19:52.171797 9908 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.60:38323
I20260928 20:19:52.171838 9975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.60:38323 every 8 connection(s)
I20260928 20:19:52.172245 9908 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/master-2/data/info.pb
I20260928 20:19:52.173606 9976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:52.175485 9976 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.18.62" port: 42379 } has no permanent_uuid. Determining permanent_uuid...
I20260928 20:19:52.181304 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 9908
Sep 28 20:19:52 dist-test-slave-b83v krb5kdc[9212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626792, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.18.60@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM
I20260928 20:19:52.187305 9315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:52.176601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35244 (local address 127.3.18.62:42379)
0928 20:19:52.177427 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:52.177429 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:52.177441 (+ 12us) server_negotiation.cc:407] Connection header received
0928 20:19:52.177623 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:52.177624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:52.177634 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:52.177703 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:52.178375 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.179005 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.180305 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.180575 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.180644 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:52.183551 (+ 2907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:52.183553 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:52.183554 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:52.183563 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:52.185339 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:52.185990 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:52.185992 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:52.185993 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:52.186070 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:52.186433 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:52.186434 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:52.186434 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:52.186599 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:52.186682 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:52.187103 (+ 421us) server_negotiation.cc:299] Negotiation successful
0928 20:19:52.187212 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":800}
I20260928 20:19:52.187583 9978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:52.176466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:42379 (local address 127.0.0.1:35244)
0928 20:19:52.176671 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:52.176717 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:52.177498 (+ 781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:52.177766 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:52.177770 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:52.178036 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:52.178238 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:52.178241 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.179135 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.179136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:52.180039 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:52.180042 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.180722 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.180723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:52.180854 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:52.181536 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:19:52.181548 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:52.183450 (+ 1902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:52.185494 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:52.185502 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:52.185505 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:52.185853 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:52.186207 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:52.186208 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:52.186208 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:52.186333 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:52.186715 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:52.186722 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:52.186915 (+ 193us) client_negotiation.cc:769] Sending connection context
0928 20:19:52.187076 (+ 161us) client_negotiation.cc:240] Negotiation successful
0928 20:19:52.187250 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":103,"threads_started":1}
I20260928 20:19:52.188930 9976 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.18.61" port: 33799 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 20:19:52 dist-test-slave-b83v krb5kdc[9212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.18.60@KRBTEST.COM
I20260928 20:19:52.193640 9979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:52.182070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60282 (local address 127.3.18.60:38323)
0928 20:19:52.182238 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:52.182243 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:52.182268 (+ 25us) server_negotiation.cc:407] Connection header received
0928 20:19:52.182288 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:52.182290 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:52.182307 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:52.182481 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:52.183235 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.184797 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.185933 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.186578 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.186720 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:52.189848 (+ 3128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:52.189857 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:52.189857 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:52.189865 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:52.191733 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:52.192282 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:52.192286 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:52.192286 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:52.192390 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:52.192792 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:52.192793 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:52.192794 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:52.193027 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:52.193198 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:52.193436 (+ 238us) server_negotiation.cc:299] Negotiation successful
0928 20:19:52.193531 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":105,"threads_started":1}
Sep 28 20:19:52 dist-test-slave-b83v krb5kdc[9212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626792, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.18.60@KRBTEST.COM for oryx/127.3.18.61@KRBTEST.COM
I20260928 20:19:52.198257 9978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:52.189236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.61:33799 (local address 127.0.0.1:57448)
0928 20:19:52.189295 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:52.189307 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:52.189354 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:52.189519 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:52.189520 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:52.189676 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:52.189745 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:52.189747 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.190791 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.190792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:52.191670 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:52.191673 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.192354 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.192356 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:52.192445 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:52.192873 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:19:52.192878 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:52.194619 (+ 1741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:52.196809 (+ 2190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:52.196811 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:52.196812 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:52.197144 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:52.197415 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:52.197416 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:52.197416 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:52.197498 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:52.197839 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:52.197840 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:52.197896 (+ 56us) client_negotiation.cc:769] Sending connection context
0928 20:19:52.198004 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 20:19:52.198135 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 20:19:52.198271 9905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:52.189362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57448 (local address 127.3.18.61:33799)
0928 20:19:52.189405 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:52.189407 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:52.189414 (+ 7us) server_negotiation.cc:407] Connection header received
0928 20:19:52.189438 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:52.189438 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:52.189449 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:52.189497 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:52.189855 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.190672 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.191881 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.192263 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.192318 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:52.194711 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:52.194713 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:52.194713 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:52.194723 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:52.196642 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:52.197237 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:52.197239 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:52.197240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:52.197327 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:52.197578 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:52.197580 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:52.197580 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:52.197733 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:52.197847 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:52.198035 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 20:19:52.198138 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 20:19:52.198954 9976 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.18.60" port: 38323 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 20:19:52 dist-test-slave-b83v krb5kdc[9212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626792, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.18.60@KRBTEST.COM for oryx/127.3.18.60@KRBTEST.COM
I20260928 20:19:52.206766 9978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:52.199467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.60:38323 (local address 127.0.0.1:60296)
0928 20:19:52.199575 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:52.199590 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:52.199642 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:52.199865 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:52.199866 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:52.200017 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:52.200054 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:52.200055 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.200739 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.200740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:52.201234 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:52.201235 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.201610 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.201610 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:52.201658 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:52.202146 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:19:52.202150 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:52.203525 (+ 1375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:52.205504 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:52.205505 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:52.205505 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:52.205695 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:52.205993 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:52.205994 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:52.205995 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:52.206067 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:52.206464 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:52.206465 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:52.206510 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 20:19:52.206583 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 20:19:52.206673 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260928 20:19:52.206820 9979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:52.199569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60296 (local address 127.3.18.60:38323)
0928 20:19:52.199619 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:52.199621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:52.199629 (+ 8us) server_negotiation.cc:407] Connection header received
0928 20:19:52.199700 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:52.199701 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:52.199712 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:52.199857 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:52.200140 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.200631 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.201319 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.201526 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.201615 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:52.203621 (+ 2006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:52.203622 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:52.203623 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:52.203629 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:52.205324 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:52.205836 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:52.205837 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:52.205837 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:52.205896 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:52.206182 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:52.206183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:52.206183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:52.206309 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:52.206382 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:52.206605 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 20:19:52.206721 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":35}
I20260928 20:19:52.208500 9976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 645a6ba8e676428585dc367ef37372c4: Bootstrap starting.
I20260928 20:19:52.209415 9976 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 645a6ba8e676428585dc367ef37372c4: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:52.209728 9976 log.cc:824] T 00000000000000000000000000000000 P 645a6ba8e676428585dc367ef37372c4: Log is configured to *not* fsync() on all Append() calls
I20260928 20:19:52.210677 9976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 645a6ba8e676428585dc367ef37372c4: No bootstrap required, opened a new log
I20260928 20:19:52.212293 9976 raft_consensus.cc:348] T 00000000000000000000000000000000 P 645a6ba8e676428585dc367ef37372c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef4554cccf754a53a7d74f685421731a" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 42379 } } peers { permanent_uuid: "d13b62b954784d05bd09150e14f708b7" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 33799 } } peers { permanent_uuid: "645a6ba8e676428585dc367ef37372c4" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 38323 } }
I20260928 20:19:52.212476 9976 raft_consensus.cc:374] T 00000000000000000000000000000000 P 645a6ba8e676428585dc367ef37372c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:52.212527 9976 raft_consensus.cc:729] T 00000000000000000000000000000000 P 645a6ba8e676428585dc367ef37372c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 645a6ba8e676428585dc367ef37372c4, State: Initialized, Role: FOLLOWER
I20260928 20:19:52.212690 9976 consensus_queue.cc:260] T 00000000000000000000000000000000 P 645a6ba8e676428585dc367ef37372c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef4554cccf754a53a7d74f685421731a" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 42379 } } peers { permanent_uuid: "d13b62b954784d05bd09150e14f708b7" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 33799 } } peers { permanent_uuid: "645a6ba8e676428585dc367ef37372c4" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 38323 } }
I20260928 20:19:52.213028 9976 sys_catalog.cc:564] T 00000000000000000000000000000000 P 645a6ba8e676428585dc367ef37372c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:19:52.213024 9981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 645a6ba8e676428585dc367ef37372c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef4554cccf754a53a7d74f685421731a" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 42379 } } peers { permanent_uuid: "d13b62b954784d05bd09150e14f708b7" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 33799 } } peers { permanent_uuid: "645a6ba8e676428585dc367ef37372c4" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 38323 } } }
I20260928 20:19:52.213135 9981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 645a6ba8e676428585dc367ef37372c4 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 20:19:52.215164 9992 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 645a6ba8e676428585dc367ef37372c4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 20:19:52.215243 9992 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 20:19:52.218060 9908 master_runner.cc:428] Detected that this master 645a6ba8e676428585dc367ef37372c4 is joining an existing cluster
I20260928 20:19:52.218110 9908 master_runner.cc:432] Initiating AddMaster RPC to add 127.3.18.60:38323
I20260928 20:19:52.226341 9979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:52.219156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60312 (local address 127.3.18.60:38323)
0928 20:19:52.219198 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:52.219201 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:52.219509 (+ 308us) server_negotiation.cc:407] Connection header received
0928 20:19:52.219598 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:52.219599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:52.219608 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:52.219637 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:52.220235 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.221001 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.222122 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.222401 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.222440 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:52.223616 (+ 1176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:52.223618 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:52.223618 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:52.223625 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:52.225133 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:52.225487 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:52.225488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:52.225489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:52.225541 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:52.225751 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:52.225753 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:52.225753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:52.225877 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:52.225967 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:52.226125 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 20:19:52.226265 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":21}
I20260928 20:19:52.226348 10000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:52.219257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.60:38323 (local address 127.0.0.1:60312)
0928 20:19:52.219440 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:52.219449 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:52.219529 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:52.219696 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:52.219697 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:52.220055 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:52.220151 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:52.220153 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.221092 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.221092 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:52.222029 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:52.222033 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.222535 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.222536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:52.222594 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:52.223143 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:19:52.223148 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:52.223507 (+ 359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:52.225229 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:52.225229 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:52.225230 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:52.225397 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:52.225627 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:52.225627 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:52.225628 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:52.225682 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:52.225950 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:52.225951 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:52.226005 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 20:19:52.226103 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 20:19:52.226216 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":67,"thread_start_us":64,"threads_started":1}
I20260928 20:19:52.245225 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 9319
I20260928 20:19:52.249264 9905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:52.219210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57464 (local address 127.3.18.61:33799)
0928 20:19:52.219256 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:52.219258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:52.219614 (+ 356us) server_negotiation.cc:407] Connection header received
0928 20:19:52.219710 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:52.219712 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:52.219725 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:52.219755 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:52.220243 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.221002 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.222148 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.222451 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.222495 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:52.245191 (+ 22696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:52.245193 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:52.245193 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:52.245203 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:52.247884 (+ 2681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:52.248334 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:52.248336 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:52.248337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:52.248396 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:52.248588 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:52.248589 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:52.248589 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:52.248737 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:52.248777 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:52.249018 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 20:19:52.249112 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 20:19:52.249369 9315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:52.219339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35254 (local address 127.3.18.62:42379)
0928 20:19:52.219391 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:52.219393 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:52.219476 (+ 83us) server_negotiation.cc:407] Connection header received
0928 20:19:52.219533 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:52.219534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:52.219565 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:52.219627 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:52.220255 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.221021 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.222163 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.222459 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.222502 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:52.245666 (+ 23164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:52.245669 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:52.245669 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:52.245678 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:52.248163 (+ 2485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:52.248559 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:52.248560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:52.248561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:52.248619 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:52.248807 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:52.248808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:52.248809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:52.248954 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:52.248995 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:52.249190 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 20:19:52.249271 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 20:19:52.249699 9999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:52.219305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.61:33799 (local address 127.0.0.1:57464)
0928 20:19:52.219561 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:52.219570 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:52.219637 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:52.219857 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:52.219857 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:52.220092 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:52.220151 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:52.220153 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.221092 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.221093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:52.222056 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:52.222058 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.222566 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.222566 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:52.222620 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:52.244469 (+ 21849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:19:52.244476 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:52.245031 (+ 555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:52.247995 (+ 2964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:52.247996 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:52.247997 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:52.248241 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:52.248465 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:52.248465 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:52.248466 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:52.248526 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:52.248820 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:52.248821 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:52.248897 (+ 76us) client_negotiation.cc:769] Sending connection context
0928 20:19:52.249395 (+ 498us) client_negotiation.cc:240] Negotiation successful
0928 20:19:52.249528 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":26,"thread_start_us":72,"threads_started":1}
I20260928 20:19:52.250550 9998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:52.219127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:42379 (local address 127.0.0.1:35254)
0928 20:19:52.219386 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:52.219398 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:52.219470 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:52.219681 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:52.219682 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:52.220058 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:52.220154 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:52.220156 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.221116 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.221117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:52.222060 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:52.222062 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.222582 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.222583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:52.222648 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:52.244768 (+ 22120us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:19:52.244774 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:52.245541 (+ 767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:52.248262 (+ 2721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:52.248263 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:52.248264 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:52.248474 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:52.248688 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:52.248689 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:52.248689 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:52.248742 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:52.249042 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:52.249043 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:52.249084 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 20:19:52.250318 (+ 1234us) client_negotiation.cc:240] Negotiation successful
0928 20:19:52.250433 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":2,"thread_start_us":87,"threads_started":1}
I20260928 20:19:52.251981 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.1:44945
--local_ip_for_outbound_sockets=127.3.18.1
--tserver_master_addrs=127.3.18.62:42379
--webserver_port=42685
--webserver_interface=127.3.18.1
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/krb5kdc/oryx_127.3.18.1.keytab
--principal=oryx/127.3.18.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:45321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.18.62:42379,127.3.18.61:33799
--tserver_master_addrs=127.3.18.62:42379,127.3.18.61:33799,127.3.18.60:38323 with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 20:19:52 dist-test-slave-b83v krb5kdc[9212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626791, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.18.62@KRBTEST.COM for oryx/127.3.18.60@KRBTEST.COM
I20260928 20:19:52.261736 9979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:52.253241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60322 (local address 127.3.18.60:38323)
0928 20:19:52.253468 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:52.253469 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:52.253475 (+ 6us) server_negotiation.cc:407] Connection header received
0928 20:19:52.253502 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:52.253503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:52.253510 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:52.253577 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:52.253917 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.254676 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.255861 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.256167 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.256236 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:52.258651 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:52.258653 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:52.258653 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:52.258660 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:52.260395 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:52.260871 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:52.260873 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:52.260873 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:52.260936 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:52.261159 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:52.261160 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:52.261160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:52.261297 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:52.261365 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:52.261555 (+ 190us) server_negotiation.cc:299] Negotiation successful
0928 20:19:52.261636 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207}
I20260928 20:19:52.261785 9814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:52.253141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.60:38323 (local address 127.0.0.1:60322)
0928 20:19:52.253232 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:52.253243 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:52.253283 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:52.253583 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:52.253584 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:52.253775 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:52.253830 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:52.253832 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.254810 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.254811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:52.255689 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:52.255691 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.256291 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.256292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:52.256362 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:52.256839 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:19:52.256845 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:52.258540 (+ 1695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:52.260517 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:52.260518 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:52.260519 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:52.260781 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:52.261016 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:52.261017 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:52.261018 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:52.261089 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:52.261370 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:52.261371 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:52.261430 (+ 59us) client_negotiation.cc:769] Sending connection context
0928 20:19:52.261521 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 20:19:52.261644 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260928 20:19:52.262735 9257 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.3.18.60:38323: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "645a6ba8e676428585dc367ef37372c4"
member_type: NON_VOTER
last_known_addr {
host: "127.3.18.60"
port: 38323
}
attrs {
promote: true
}
}
dest_uuid: "ef4554cccf754a53a7d74f685421731a"
cas_config_opid_index: 5
I20260928 20:19:52.263015 9257 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef4554cccf754a53a7d74f685421731a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ef4554cccf754a53a7d74f685421731a" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 42379 } } peers { permanent_uuid: "d13b62b954784d05bd09150e14f708b7" member_type: NON_VOTER last_known_addr { host: "127.3.18.61" port: 33799 } attrs { promote: true } } peers { permanent_uuid: "645a6ba8e676428585dc367ef37372c4" member_type: NON_VOTER last_known_addr { host: "127.3.18.60" port: 38323 } attrs { promote: true } }
I20260928 20:19:52.263458 9816 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ef4554cccf754a53a7d74f685421731a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 645a6ba8e676428585dc367ef37372c4 (127.3.18.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ef4554cccf754a53a7d74f685421731a" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 42379 } } peers { permanent_uuid: "d13b62b954784d05bd09150e14f708b7" member_type: NON_VOTER last_known_addr { host: "127.3.18.61" port: 33799 } attrs { promote: true } } peers { permanent_uuid: "645a6ba8e676428585dc367ef37372c4" member_type: NON_VOTER last_known_addr { host: "127.3.18.60" port: 38323 } attrs { promote: true } } }
I20260928 20:19:52.264503 9816 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.3.18.60:38323
I20260928 20:19:52.264578 9816 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef4554cccf754a53a7d74f685421731a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ef4554cccf754a53a7d74f685421731a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ef4554cccf754a53a7d74f685421731a" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 42379 } } peers { permanent_uuid: "d13b62b954784d05bd09150e14f708b7" member_type: NON_VOTER last_known_addr { host: "127.3.18.61" port: 33799 } attrs { promote: true } } peers { permanent_uuid: "645a6ba8e676428585dc367ef37372c4" member_type: NON_VOTER last_known_addr { host: "127.3.18.60" port: 38323 } attrs { promote: true } } }
I20260928 20:19:52.264653 9816 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef4554cccf754a53a7d74f685421731a [sys.catalog]: This master's current role is: LEADER
I20260928 20:19:52.265239 9908 master.cc:537] Master@127.3.18.60:38323 shutting down...
I20260928 20:19:52.265635 9950 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 645a6ba8e676428585dc367ef37372c4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:52.266638 9950 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 645a6ba8e676428585dc367ef37372c4 [term 1 FOLLOWER]: Refusing update from remote peer ef4554cccf754a53a7d74f685421731a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 20:19:52.267100 9815 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ef4554cccf754a53a7d74f685421731a [LEADER]: Connected to new peer: Peer: permanent_uuid: "645a6ba8e676428585dc367ef37372c4" member_type: NON_VOTER last_known_addr { host: "127.3.18.60" port: 38323 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 20:19:52.267830 9244 consensus_peers.cc:487] T 00000000000000000000000000000000 P ef4554cccf754a53a7d74f685421731a -> Peer 645a6ba8e676428585dc367ef37372c4 (127.3.18.60:38323): Couldn't send request to peer 645a6ba8e676428585dc367ef37372c4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 20:19:52.269441 9905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:52.265035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57474 (local address 127.3.18.61:33799)
0928 20:19:52.265080 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:52.265083 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:52.265605 (+ 522us) server_negotiation.cc:407] Connection header received
0928 20:19:52.265712 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:52.265713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:52.265737 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:52.265778 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:19:52.266008 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.266766 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.268264 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.268878 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.268926 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:52.269002 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:52.269310 (+ 308us) server_negotiation.cc:299] Negotiation successful
0928 20:19:52.269360 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 20:19:52.269630 9814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:52.264958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.61:33799 (local address 127.0.0.1:57474)
0928 20:19:52.265545 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:52.265556 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:52.265636 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:52.265884 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:52.265885 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:52.265887 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:19:52.265931 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:52.265932 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.266879 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.266880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:52.268157 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:52.268160 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.269136 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.269137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:52.269204 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:52.269229 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 20:19:52.269518 (+ 289us) client_negotiation.cc:240] Negotiation successful
0928 20:19:52.269567 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569}
I20260928 20:19:52.269858 9864 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d13b62b954784d05bd09150e14f708b7 [term 1 LEARNER]: Refusing update from remote peer ef4554cccf754a53a7d74f685421731a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 20:19:52.270129 9815 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ef4554cccf754a53a7d74f685421731a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13b62b954784d05bd09150e14f708b7" member_type: NON_VOTER last_known_addr { host: "127.3.18.61" port: 33799 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 20:19:52.270761 9864 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d13b62b954784d05bd09150e14f708b7 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 645a6ba8e676428585dc367ef37372c4 (127.3.18.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ef4554cccf754a53a7d74f685421731a" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 42379 } } peers { permanent_uuid: "d13b62b954784d05bd09150e14f708b7" member_type: NON_VOTER last_known_addr { host: "127.3.18.61" port: 33799 } attrs { promote: true } } peers { permanent_uuid: "645a6ba8e676428585dc367ef37372c4" member_type: NON_VOTER last_known_addr { host: "127.3.18.60" port: 38323 } attrs { promote: true } } }
I20260928 20:19:52.272135 9893 sys_catalog.cc:455] T 00000000000000000000000000000000 P d13b62b954784d05bd09150e14f708b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef4554cccf754a53a7d74f685421731a. Latest consensus state: current_term: 1 leader_uuid: "ef4554cccf754a53a7d74f685421731a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ef4554cccf754a53a7d74f685421731a" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 42379 } } peers { permanent_uuid: "d13b62b954784d05bd09150e14f708b7" member_type: NON_VOTER last_known_addr { host: "127.3.18.61" port: 33799 } attrs { promote: true } } peers { permanent_uuid: "645a6ba8e676428585dc367ef37372c4" member_type: NON_VOTER last_known_addr { host: "127.3.18.60" port: 38323 } attrs { promote: true } } }
I20260928 20:19:52.272238 9893 sys_catalog.cc:458] T 00000000000000000000000000000000 P d13b62b954784d05bd09150e14f708b7 [sys.catalog]: This master's current role is: LEARNER
I20260928 20:19:52.272284 9893 sys_catalog.cc:455] T 00000000000000000000000000000000 P d13b62b954784d05bd09150e14f708b7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ef4554cccf754a53a7d74f685421731a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ef4554cccf754a53a7d74f685421731a" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 42379 } } peers { permanent_uuid: "d13b62b954784d05bd09150e14f708b7" member_type: NON_VOTER last_known_addr { host: "127.3.18.61" port: 33799 } attrs { promote: true } } peers { permanent_uuid: "645a6ba8e676428585dc367ef37372c4" member_type: NON_VOTER last_known_addr { host: "127.3.18.60" port: 38323 } attrs { promote: true } } }
I20260928 20:19:52.272313 9815 raft_consensus.cc:1050] T 00000000000000000000000000000000 P ef4554cccf754a53a7d74f685421731a: attempting to promote NON_VOTER d13b62b954784d05bd09150e14f708b7 to VOTER
I20260928 20:19:52.272336 9893 sys_catalog.cc:458] T 00000000000000000000000000000000 P d13b62b954784d05bd09150e14f708b7 [sys.catalog]: This master's current role is: LEARNER
I20260928 20:19:52.272529 9815 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef4554cccf754a53a7d74f685421731a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ef4554cccf754a53a7d74f685421731a" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 42379 } } peers { permanent_uuid: "d13b62b954784d05bd09150e14f708b7" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 33799 } attrs { promote: false } } peers { permanent_uuid: "645a6ba8e676428585dc367ef37372c4" member_type: NON_VOTER last_known_addr { host: "127.3.18.60" port: 38323 } attrs { promote: true } }
W20260928 20:19:52.273370 9244 consensus_peers.cc:487] T 00000000000000000000000000000000 P ef4554cccf754a53a7d74f685421731a -> Peer 645a6ba8e676428585dc367ef37372c4 (127.3.18.60:38323): Couldn't send request to peer 645a6ba8e676428585dc367ef37372c4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 20:19:52.273406 9908 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 645a6ba8e676428585dc367ef37372c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:19:52.273473 9864 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d13b62b954784d05bd09150e14f708b7 [term 1 LEARNER]: Refusing update from remote peer ef4554cccf754a53a7d74f685421731a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 20:19:52.273592 9908 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 645a6ba8e676428585dc367ef37372c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:19:52.273640 9908 tablet_replica.cc:330] T 00000000000000000000000000000000 P 645a6ba8e676428585dc367ef37372c4: stopping tablet replica
I20260928 20:19:52.273626 9816 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ef4554cccf754a53a7d74f685421731a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13b62b954784d05bd09150e14f708b7" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 33799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260928 20:19:52.274171 9816 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ef4554cccf754a53a7d74f685421731a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d13b62b954784d05bd09150e14f708b7 (127.3.18.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ef4554cccf754a53a7d74f685421731a" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 42379 } } peers { permanent_uuid: "d13b62b954784d05bd09150e14f708b7" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 33799 } attrs { promote: false } } peers { permanent_uuid: "645a6ba8e676428585dc367ef37372c4" member_type: NON_VOTER last_known_addr { host: "127.3.18.60" port: 38323 } attrs { promote: true } } }
I20260928 20:19:52.275063 9816 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef4554cccf754a53a7d74f685421731a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ef4554cccf754a53a7d74f685421731a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ef4554cccf754a53a7d74f685421731a" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 42379 } } peers { permanent_uuid: "d13b62b954784d05bd09150e14f708b7" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 33799 } attrs { promote: false } } peers { permanent_uuid: "645a6ba8e676428585dc367ef37372c4" member_type: NON_VOTER last_known_addr { host: "127.3.18.60" port: 38323 } attrs { promote: true } } }
I20260928 20:19:52.275138 9816 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef4554cccf754a53a7d74f685421731a [sys.catalog]: This master's current role is: LEADER
I20260928 20:19:52.275195 9816 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef4554cccf754a53a7d74f685421731a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ef4554cccf754a53a7d74f685421731a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ef4554cccf754a53a7d74f685421731a" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 42379 } } peers { permanent_uuid: "d13b62b954784d05bd09150e14f708b7" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 33799 } attrs { promote: false } } peers { permanent_uuid: "645a6ba8e676428585dc367ef37372c4" member_type: NON_VOTER last_known_addr { host: "127.3.18.60" port: 38323 } attrs { promote: true } } }
I20260928 20:19:52.275235 9816 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef4554cccf754a53a7d74f685421731a [sys.catalog]: This master's current role is: LEADER
I20260928 20:19:52.275923 9864 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d13b62b954784d05bd09150e14f708b7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d13b62b954784d05bd09150e14f708b7 (127.3.18.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ef4554cccf754a53a7d74f685421731a" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 42379 } } peers { permanent_uuid: "d13b62b954784d05bd09150e14f708b7" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 33799 } attrs { promote: false } } peers { permanent_uuid: "645a6ba8e676428585dc367ef37372c4" member_type: NON_VOTER last_known_addr { host: "127.3.18.60" port: 38323 } attrs { promote: true } } }
I20260928 20:19:52.276795 10003 sys_catalog.cc:455] T 00000000000000000000000000000000 P d13b62b954784d05bd09150e14f708b7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ef4554cccf754a53a7d74f685421731a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ef4554cccf754a53a7d74f685421731a" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 42379 } } peers { permanent_uuid: "d13b62b954784d05bd09150e14f708b7" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 33799 } attrs { promote: false } } peers { permanent_uuid: "645a6ba8e676428585dc367ef37372c4" member_type: NON_VOTER last_known_addr { host: "127.3.18.60" port: 38323 } attrs { promote: true } } }
I20260928 20:19:52.276932 10003 sys_catalog.cc:458] T 00000000000000000000000000000000 P d13b62b954784d05bd09150e14f708b7 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 20:19:52.288450 9908 master.cc:552] Master@127.3.18.60:38323 shutdown complete.
I20260928 20:19:52.288522 9908 master_runner.cc:305] Clearing existing system tablet
I20260928 20:19:52.288678 9908 fs_manager.cc:359] Metadata directory not provided
I20260928 20:19:52.288731 9908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:19:52.289528 9908 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 20:19:52.290077 10006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:52.290225 9908 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:19:52.290282 9908 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 20:19:52.290304 9908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/master-2/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/master-2/wal
uuid: "645a6ba8e676428585dc367ef37372c4"
format_stamp: "Formatted at 2026-09-28 20:19:52 on dist-test-slave-b83v"
server_key: "973fb86d2949f05fc10c49e10ad8e6d8"
server_key_iv: "83e6f11a0d30707ce2c217739fc83d9b"
server_key_version: "encryptionkey@0"
I20260928 20:19:52.291023 9908 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 645a6ba8e676428585dc367ef37372c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:19:52.292768 9908 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 645a6ba8e676428585dc367ef37372c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 20:19:52.292850 9908 log.cc:1196] T 00000000000000000000000000000000 P 645a6ba8e676428585dc367ef37372c4: Deleting WAL directory at /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 20:19:52.293041 9908 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 645a6ba8e676428585dc367ef37372c4: Deleting consensus metadata
I20260928 20:19:52.293169 9908 master_runner.cc:315] Copying system tablet from 127.3.18.62:42379
I20260928 20:19:52.293784 9908 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 645a6ba8e676428585dc367ef37372c4: tablet copy: Beginning tablet copy session from remote peer at address 127.3.18.62:42379
I20260928 20:19:52.301695 9315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:52.294171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35264 (local address 127.3.18.62:42379)
0928 20:19:52.294218 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:52.294220 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:52.294289 (+ 69us) server_negotiation.cc:407] Connection header received
0928 20:19:52.294360 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:52.294361 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:52.294373 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:52.294449 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:52.294861 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.295644 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.296755 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.297060 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.297383 (+ 323us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:52.298340 (+ 957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:52.298342 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:52.298342 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:52.298350 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:52.300172 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:52.300614 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:52.300616 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:52.300616 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:52.300692 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:52.300959 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:52.300960 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:52.300960 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:52.301130 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:52.301185 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:52.301436 (+ 251us) server_negotiation.cc:299] Negotiation successful
0928 20:19:52.301560 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 20:19:52.302109 10012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:52.294093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:42379 (local address 127.0.0.1:35264)
0928 20:19:52.294227 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:52.294240 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:52.294304 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:52.294476 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:52.294477 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:52.294699 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:52.294770 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:52.294772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.295750 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.295752 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:52.296649 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:52.296653 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.297174 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.297176 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:52.297253 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:52.297793 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:19:52.297798 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:52.298240 (+ 442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:52.300271 (+ 2031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:52.300273 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:52.300273 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:52.300518 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:52.300792 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:52.300794 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:52.300794 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:52.300874 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:52.301244 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:52.301245 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:52.301306 (+ 61us) client_negotiation.cc:769] Sending connection context
0928 20:19:52.301868 (+ 562us) client_negotiation.cc:240] Negotiation successful
0928 20:19:52.302008 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":72,"threads_started":1}
I20260928 20:19:52.302394 9277 tablet_copy_service.cc:140] P ef4554cccf754a53a7d74f685421731a: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 645a6ba8e676428585dc367ef37372c4 ({username='oryx', principal='oryx/127.3.18.60@KRBTEST.COM'} at 127.0.0.1:35264)
I20260928 20:19:52.302456 9277 tablet_copy_service.cc:161] P ef4554cccf754a53a7d74f685421731a: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 645a6ba8e676428585dc367ef37372c4 at {username='oryx', principal='oryx/127.3.18.60@KRBTEST.COM'} at 127.0.0.1:35264: session id = 645a6ba8e676428585dc367ef37372c4-00000000000000000000000000000000
I20260928 20:19:52.303074 9277 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P ef4554cccf754a53a7d74f685421731a: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 20:19:52.303658 9908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 20:19:52.305377 9908 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 645a6ba8e676428585dc367ef37372c4: tablet copy: Starting download of 0 data blocks...
I20260928 20:19:52.305558 9908 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 645a6ba8e676428585dc367ef37372c4: tablet copy: Starting download of 1 WAL segments...
I20260928 20:19:52.306504 9908 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 645a6ba8e676428585dc367ef37372c4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 20:19:52.307610 9277 tablet_copy_service.cc:342] P ef4554cccf754a53a7d74f685421731a: Request end of tablet copy session 645a6ba8e676428585dc367ef37372c4-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.3.18.60@KRBTEST.COM'} at 127.0.0.1:35264
I20260928 20:19:52.307674 9277 tablet_copy_service.cc:433] P ef4554cccf754a53a7d74f685421731a: ending tablet copy session 645a6ba8e676428585dc367ef37372c4-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 645a6ba8e676428585dc367ef37372c4
I20260928 20:19:52.310997 9908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:19:52.312425 10013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:52.312685 10016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:52.312922 9908 server_base.cc:1034] running on GCE node
W20260928 20:19:52.316319 10014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:52.316506 9908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:19:52.316669 9908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:19:52.317821 9908 hybrid_clock.cc:648] HybridClock initialized: now 1790626792317792 us; error 46 us; skew 500 ppm
Sep 28 20:19:52 dist-test-slave-b83v krb5kdc[9212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626792, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.18.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:19:52.319527 9908 init.cc:377] Logged in from keytab as oryx/127.3.18.60@KRBTEST.COM (short username oryx)
I20260928 20:19:52.319826 9908 webserver.cc:460] Webserver started at http://127.3.18.60:36241/ using document root <none> and password file <none>
I20260928 20:19:52.319944 9908 fs_manager.cc:359] Metadata directory not provided
I20260928 20:19:52.319970 9908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:19:52.320744 9908 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:19:52.321638 10023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:52.321838 9908 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:19:52.321892 9908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/master-2/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/master-2/wal
uuid: "645a6ba8e676428585dc367ef37372c4"
format_stamp: "Formatted at 2026-09-28 20:19:52 on dist-test-slave-b83v"
server_key: "973fb86d2949f05fc10c49e10ad8e6d8"
server_key_iv: "83e6f11a0d30707ce2c217739fc83d9b"
server_key_version: "encryptionkey@0"
I20260928 20:19:52.321947 9908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:19:52.331597 9908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:19:52.333666 9908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:19:52.336951 9908 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.60:38323
I20260928 20:19:52.336998 10075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.60:38323 every 8 connection(s)
I20260928 20:19:52.337515 9908 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/master-2/data/info.pb
I20260928 20:19:52.338654 10076 sys_catalog.cc:263] Verifying existing consensus state
I20260928 20:19:52.339201 10076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 645a6ba8e676428585dc367ef37372c4: Bootstrap starting.
I20260928 20:19:52.342448 10076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 645a6ba8e676428585dc367ef37372c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 20:19:52.342680 10076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 645a6ba8e676428585dc367ef37372c4: Bootstrap complete.
I20260928 20:19:52.342904 10076 raft_consensus.cc:348] T 00000000000000000000000000000000 P 645a6ba8e676428585dc367ef37372c4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ef4554cccf754a53a7d74f685421731a" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 42379 } } peers { permanent_uuid: "d13b62b954784d05bd09150e14f708b7" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 33799 } attrs { promote: false } } peers { permanent_uuid: "645a6ba8e676428585dc367ef37372c4" member_type: NON_VOTER last_known_addr { host: "127.3.18.60" port: 38323 } attrs { promote: true } }
I20260928 20:19:52.343007 10076 raft_consensus.cc:729] T 00000000000000000000000000000000 P 645a6ba8e676428585dc367ef37372c4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 645a6ba8e676428585dc367ef37372c4, State: Initialized, Role: LEARNER
I20260928 20:19:52.343047 10076 consensus_queue.cc:260] T 00000000000000000000000000000000 P 645a6ba8e676428585dc367ef37372c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ef4554cccf754a53a7d74f685421731a" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 42379 } } peers { permanent_uuid: "d13b62b954784d05bd09150e14f708b7" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 33799 } attrs { promote: false } } peers { permanent_uuid: "645a6ba8e676428585dc367ef37372c4" member_type: NON_VOTER last_known_addr { host: "127.3.18.60" port: 38323 } attrs { promote: true } }
I20260928 20:19:52.343236 10076 sys_catalog.cc:564] T 00000000000000000000000000000000 P 645a6ba8e676428585dc367ef37372c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:19:52.343266 10079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 645a6ba8e676428585dc367ef37372c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ef4554cccf754a53a7d74f685421731a" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 42379 } } peers { permanent_uuid: "d13b62b954784d05bd09150e14f708b7" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 33799 } attrs { promote: false } } peers { permanent_uuid: "645a6ba8e676428585dc367ef37372c4" member_type: NON_VOTER last_known_addr { host: "127.3.18.60" port: 38323 } attrs { promote: true } } }
I20260928 20:19:52.343495 10079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 645a6ba8e676428585dc367ef37372c4 [sys.catalog]: This master's current role is: LEARNER
I20260928 20:19:52.345867 10090 catalog_manager.cc:1259] Loaded cluster ID: 44e10b6ab55043b4b4c53ad9882d8ab4
I20260928 20:19:52.345983 10090 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 645a6ba8e676428585dc367ef37372c4: loading cluster ID for follower catalog manager: success
I20260928 20:19:52.347239 10090 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 645a6ba8e676428585dc367ef37372c4: acquiring CA information for follower catalog manager: success
I20260928 20:19:52.347640 10090 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 645a6ba8e676428585dc367ef37372c4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 20:19:52.362484 10001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:19:52.362759 10001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:19:52.362792 10001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:19:52.364362 10001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:19:52.364410 10001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:19:52.364440 10001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.1
I20260928 20:19:52.366176 10001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:45321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/krb5kdc/oryx_127.3.18.1.keytab
--principal=oryx/127.3.18.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.1:44945
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.18.1
--webserver_port=42685
--tserver_master_addrs=127.3.18.62:42379,127.3.18.61:33799,127.3.18.60:38323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.1
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:19:52.366474 10001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:19:52.366756 10001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:19:52.373257 10095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:52.373557 10096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:52.373289 10098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:52.373710 10001 server_base.cc:1034] running on GCE node
I20260928 20:19:52.373927 10001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:19:52.374239 10001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:19:52.375407 10001 hybrid_clock.cc:648] HybridClock initialized: now 1790626792375379 us; error 48 us; skew 500 ppm
Sep 28 20:19:52 dist-test-slave-b83v krb5kdc[9212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626792, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.18.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:19:52.378302 10001 init.cc:377] Logged in from keytab as oryx/127.3.18.1@KRBTEST.COM (short username oryx)
I20260928 20:19:52.378839 10001 webserver.cc:460] Webserver started at http://127.3.18.1:42685/ using document root <none> and password file <none>
I20260928 20:19:52.379117 10001 fs_manager.cc:359] Metadata directory not provided
I20260928 20:19:52.379177 10001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:19:52.380784 10001 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:19:52.381511 10105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:52.381790 10001 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:19:52.381914 10001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-0/wal
uuid: "39e67e2773bb4225a13ec3d8425a088a"
format_stamp: "Formatted at 2026-09-28 20:19:51 on dist-test-slave-b83v"
server_key: "250c489c34064eb62142fc0050022ed4"
server_key_iv: "54d155614f6cf7c60a419ac85a843a3b"
server_key_version: "encryptionkey@0"
I20260928 20:19:52.382429 10001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:19:52.398562 10001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:19:52.401223 10001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:19:52.401403 10001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:19:52.401662 10001 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:19:52.402027 10001 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:19:52.402086 10001 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:52.402130 10001 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:19:52.402158 10001 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:52.408623 10001 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.1:44945
I20260928 20:19:52.408686 10218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.1:44945 every 8 connection(s)
I20260928 20:19:52.409036 10001 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
Sep 28 20:19:52 dist-test-slave-b83v krb5kdc[9212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626792, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.18.1@KRBTEST.COM for oryx/127.3.18.61@KRBTEST.COM
Sep 28 20:19:52 dist-test-slave-b83v krb5kdc[9212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626792, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.18.1@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM
Sep 28 20:19:52 dist-test-slave-b83v krb5kdc[9212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626792, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.18.1@KRBTEST.COM for oryx/127.3.18.60@KRBTEST.COM
I20260928 20:19:52.419370 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 10001
I20260928 20:19:52.419535 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 9453
I20260928 20:19:52.420387 9905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:52.409823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:46045 (local address 127.3.18.61:33799)
0928 20:19:52.409872 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:52.409875 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:52.410150 (+ 275us) server_negotiation.cc:407] Connection header received
0928 20:19:52.410799 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:52.410800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:52.410814 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:52.410877 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:52.411690 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.412532 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.413563 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.413854 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.413911 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:52.416714 (+ 2803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:52.416716 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:52.416716 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:52.416723 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:52.418472 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:52.419030 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:52.419031 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:52.419031 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:52.419114 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:52.419507 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:52.419508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:52.419509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:52.419669 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:52.419717 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:52.420179 (+ 462us) server_negotiation.cc:299] Negotiation successful
0928 20:19:52.420263 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 20:19:52.420495 9315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:52.409869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:49103 (local address 127.3.18.62:42379)
0928 20:19:52.409913 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:52.409915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:52.410207 (+ 292us) server_negotiation.cc:407] Connection header received
0928 20:19:52.410867 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:52.410869 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:52.410881 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:52.410952 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:52.411677 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.412237 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.413586 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.413779 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.413813 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:52.417128 (+ 3315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:52.417129 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:52.417129 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:52.417136 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:52.418860 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:52.419571 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:52.419572 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:52.419573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:52.419631 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:52.419850 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:52.419851 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:52.419851 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:52.420001 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:52.420045 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:52.420264 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 20:19:52.420351 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 20:19:52.421118 10226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:52.409811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:42379 (local address 127.3.18.1:49103)
0928 20:19:52.410155 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:52.410163 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:52.410673 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:52.410994 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:52.410999 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:52.411368 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:52.411569 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:52.411574 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.412344 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.412345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:52.413480 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:52.413482 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.413918 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.413918 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:52.414046 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:52.414751 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:19:52.414762 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:52.417042 (+ 2280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:52.419205 (+ 2163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:52.419206 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:52.419207 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:52.419472 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:52.419702 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:52.419702 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:52.419703 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:52.419784 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:52.420092 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:52.420092 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:52.420138 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 20:19:52.420583 (+ 445us) client_negotiation.cc:240] Negotiation successful
0928 20:19:52.420781 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":217,"thread_start_us":61,"threads_started":1}
I20260928 20:19:52.421298 10225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:52.410010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:50329 (local address 127.3.18.60:38323)
0928 20:19:52.410119 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:52.410121 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:52.410166 (+ 45us) server_negotiation.cc:407] Connection header received
0928 20:19:52.411012 (+ 846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:52.411013 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:52.411026 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:52.411099 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:52.411686 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.412694 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.413502 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.413802 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.413870 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:52.418114 (+ 4244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:52.418115 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:52.418115 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:52.418123 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:52.419901 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:52.420264 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:52.420265 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:52.420265 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:52.420321 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:52.420504 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:52.420505 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:52.420505 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:52.420659 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:52.420715 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:52.421098 (+ 383us) server_negotiation.cc:299] Negotiation successful
0928 20:19:52.421181 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":59,"threads_started":1}
I20260928 20:19:52.421522 10223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:52.409811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.61:33799 (local address 127.3.18.1:46045)
0928 20:19:52.410003 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:52.410055 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:52.410694 (+ 639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:52.411327 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:52.411328 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:52.411536 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:52.411587 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:52.411588 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.412618 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.412619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:52.413485 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:52.413487 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.413969 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.413970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:52.414044 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:52.414800 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:19:52.414805 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:52.416606 (+ 1801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:52.418604 (+ 1998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:52.418611 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:52.418613 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:52.418931 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:52.419201 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:52.419202 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:52.419202 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:52.419358 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:52.419833 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:52.419839 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:52.420015 (+ 176us) client_negotiation.cc:769] Sending connection context
0928 20:19:52.420959 (+ 944us) client_negotiation.cc:240] Negotiation successful
0928 20:19:52.421045 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":31,"thread_start_us":87,"threads_started":1}
I20260928 20:19:52.421615 10224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:52.409933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.60:38323 (local address 127.3.18.1:50329)
0928 20:19:52.410103 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:52.410114 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:52.410673 (+ 559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:52.411121 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:52.411122 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:52.411406 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:52.411571 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:52.411579 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.412781 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.412782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:52.413396 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:52.413398 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.413899 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.413900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:52.414046 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:52.414752 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:19:52.414762 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:52.418011 (+ 3249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:52.419984 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:52.419985 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:52.419985 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:52.420188 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:52.420385 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:52.420385 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:52.420386 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:52.420437 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:52.420738 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:52.420738 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:52.420777 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 20:19:52.420835 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 20:19:52.420928 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":60,"threads_started":1}
I20260928 20:19:52.422055 10221 heartbeater.cc:347] Connected to a master server at 127.3.18.61:33799
I20260928 20:19:52.422163 10221 heartbeater.cc:464] Registering TS with master...
I20260928 20:19:52.422348 10221 heartbeater.cc:511] Master 127.3.18.61:33799 requested a full tablet report, sending...
I20260928 20:19:52.422961 9854 ts_manager.cc:194] Registered new tserver with Master: 39e67e2773bb4225a13ec3d8425a088a (127.3.18.1:44945)
I20260928 20:19:52.425830 10219 heartbeater.cc:347] Connected to a master server at 127.3.18.60:38323
I20260928 20:19:52.425904 10219 heartbeater.cc:464] Registering TS with master...
I20260928 20:19:52.425976 10220 heartbeater.cc:347] Connected to a master server at 127.3.18.62:42379
I20260928 20:19:52.425992 10219 heartbeater.cc:511] Master 127.3.18.60:38323 requested a full tablet report, sending...
I20260928 20:19:52.426044 10220 heartbeater.cc:464] Registering TS with master...
I20260928 20:19:52.426126 10220 heartbeater.cc:511] Master 127.3.18.62:42379 requested a full tablet report, sending...
I20260928 20:19:52.426388 9257 ts_manager.cc:194] Re-registered known tserver with Master: 39e67e2773bb4225a13ec3d8425a088a (127.3.18.1:44945)
I20260928 20:19:52.426451 10040 ts_manager.cc:194] Registered new tserver with Master: 39e67e2773bb4225a13ec3d8425a088a (127.3.18.1:44945)
I20260928 20:19:52.426635 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.2:35465
--local_ip_for_outbound_sockets=127.3.18.2
--tserver_master_addrs=127.3.18.62:42379
--webserver_port=37573
--webserver_interface=127.3.18.2
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/krb5kdc/oryx_127.3.18.2.keytab
--principal=oryx/127.3.18.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:45321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.18.62:42379,127.3.18.61:33799
--tserver_master_addrs=127.3.18.62:42379,127.3.18.61:33799,127.3.18.60:38323 with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 20:19:52.426965 9257 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.18.1@KRBTEST.COM'} at 127.3.18.1:49103
W20260928 20:19:52.513630 10227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:19:52.513852 10227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:19:52.513880 10227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:19:52.515079 10227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:19:52.515120 10227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:19:52.515147 10227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.2
I20260928 20:19:52.516731 10227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:45321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/krb5kdc/oryx_127.3.18.2.keytab
--principal=oryx/127.3.18.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.2:35465
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.18.2
--webserver_port=37573
--tserver_master_addrs=127.3.18.62:42379,127.3.18.61:33799,127.3.18.60:38323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.2
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:19:52.516942 10227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:19:52.517132 10227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:19:52.522554 10232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:52.522555 10233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:52.522557 10235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:52.522783 10227 server_base.cc:1034] running on GCE node
I20260928 20:19:52.523120 10227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:19:52.523347 10227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:19:52.524655 10227 hybrid_clock.cc:648] HybridClock initialized: now 1790626792524624 us; error 61 us; skew 500 ppm
Sep 28 20:19:52 dist-test-slave-b83v krb5kdc[9212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626792, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.18.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:19:52.527297 10227 init.cc:377] Logged in from keytab as oryx/127.3.18.2@KRBTEST.COM (short username oryx)
I20260928 20:19:52.527815 10227 webserver.cc:460] Webserver started at http://127.3.18.2:37573/ using document root <none> and password file <none>
I20260928 20:19:52.528072 10227 fs_manager.cc:359] Metadata directory not provided
I20260928 20:19:52.528165 10227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:19:52.529624 10227 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 20:19:52.530376 10242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:52.530676 10227 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:19:52.530818 10227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-1/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-1/wal
uuid: "14a5194f37a543ad9cf9c8130a5d11af"
format_stamp: "Formatted at 2026-09-28 20:19:51 on dist-test-slave-b83v"
server_key: "244c25b2bda5e079d7739d4cf66d0c5e"
server_key_iv: "d770da6922462fb5921b98e0d6e1a9a5"
server_key_version: "encryptionkey@0"
I20260928 20:19:52.531272 10227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:19:52.549329 10227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:19:52.551851 10227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:19:52.552062 10227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:19:52.552333 10227 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:19:52.552755 10227 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:19:52.552821 10227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:52.552856 10227 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:19:52.552935 10227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:52.559361 10227 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.2:35465
I20260928 20:19:52.559414 10355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.2:35465 every 8 connection(s)
I20260928 20:19:52.559900 10227 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
I20260928 20:19:52.562167 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 10227
I20260928 20:19:52.562345 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 9587
Sep 28 20:19:52 dist-test-slave-b83v krb5kdc[9212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626792, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.18.2@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM
I20260928 20:19:52.569973 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.3:38189
--local_ip_for_outbound_sockets=127.3.18.3
--tserver_master_addrs=127.3.18.62:42379
--webserver_port=44589
--webserver_interface=127.3.18.3
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/krb5kdc/oryx_127.3.18.3.keytab
--principal=oryx/127.3.18.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:45321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.18.62:42379,127.3.18.61:33799
--tserver_master_addrs=127.3.18.62:42379,127.3.18.61:33799,127.3.18.60:38323 with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 20:19:52 dist-test-slave-b83v krb5kdc[9212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626792, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.18.2@KRBTEST.COM for oryx/127.3.18.60@KRBTEST.COM
Sep 28 20:19:52 dist-test-slave-b83v krb5kdc[9212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626792, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.18.2@KRBTEST.COM for oryx/127.3.18.61@KRBTEST.COM
I20260928 20:19:52.574749 9905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:52.561021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:43063 (local address 127.3.18.61:33799)
0928 20:19:52.561070 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:52.561071 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:52.561096 (+ 25us) server_negotiation.cc:407] Connection header received
0928 20:19:52.562481 (+ 1385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:52.562482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:52.562494 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:52.562521 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:52.563486 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.564436 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.566974 (+ 2538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.567744 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.568117 (+ 373us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:52.571078 (+ 2961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:52.571080 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:52.571080 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:52.571089 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:52.572843 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:52.573421 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:52.573422 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:52.573423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:52.573530 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:52.573824 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:52.573826 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:52.573827 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:52.574012 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:52.574103 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:52.574512 (+ 409us) server_negotiation.cc:299] Negotiation successful
0928 20:19:52.574625 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 20:19:52.574997 10361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:52.560690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.61:33799 (local address 127.3.18.2:43063)
0928 20:19:52.561036 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:52.561047 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:52.562215 (+ 1168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:52.562607 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:52.562612 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:52.563009 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:52.563336 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:52.563342 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.564805 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.564806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:52.566277 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:52.566281 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.568181 (+ 1900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.568183 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:52.568278 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:52.568751 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:19:52.568756 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:52.570970 (+ 2214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:52.572957 (+ 1987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:52.572967 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:52.572971 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:52.573328 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:52.573602 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:52.573603 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:52.573603 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:52.573763 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:52.574128 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:52.574135 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:52.574341 (+ 206us) client_negotiation.cc:769] Sending connection context
0928 20:19:52.574467 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 20:19:52.574703 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"mutex_wait_us":58,"thread_start_us":115,"threads_started":1}
I20260928 20:19:52.575299 9315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:52.561021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:57149 (local address 127.3.18.62:42379)
0928 20:19:52.561067 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:52.561069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:52.561080 (+ 11us) server_negotiation.cc:407] Connection header received
0928 20:19:52.562355 (+ 1275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:52.562357 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:52.562371 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:52.562405 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:52.563931 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.564690 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.566404 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.566799 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.566885 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:52.570352 (+ 3467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:52.570354 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:52.570354 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:52.570365 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:52.573408 (+ 3043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:52.573968 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:52.573970 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:52.573971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:52.574037 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:52.574292 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:52.574293 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:52.574293 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:52.574496 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:52.574561 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:52.575069 (+ 508us) server_negotiation.cc:299] Negotiation successful
0928 20:19:52.575177 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 20:19:52.575649 10358 heartbeater.cc:347] Connected to a master server at 127.3.18.61:33799
I20260928 20:19:52.575682 10225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:52.560834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:33417 (local address 127.3.18.60:38323)
0928 20:19:52.560945 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:52.560947 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:52.561305 (+ 358us) server_negotiation.cc:407] Connection header received
0928 20:19:52.562328 (+ 1023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:52.562330 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:52.562346 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:52.562378 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:52.564008 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.564744 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.567097 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.567436 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.567474 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:52.571419 (+ 3945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:52.571421 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:52.571421 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:52.571429 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:52.573401 (+ 1972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:52.574293 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:52.574295 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:52.574296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:52.574615 (+ 319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:52.575002 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:52.575003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:52.575003 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:52.575153 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:52.575219 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:52.575462 (+ 243us) server_negotiation.cc:299] Negotiation successful
0928 20:19:52.575576 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 20:19:52.575927 10358 heartbeater.cc:464] Registering TS with master...
I20260928 20:19:52.576076 10360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:52.560688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:42379 (local address 127.3.18.2:57149)
0928 20:19:52.560914 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:52.560975 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:52.562198 (+ 1223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:52.562606 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:52.562611 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:52.563711 (+ 1100us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:52.563840 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:52.563841 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.565076 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.565076 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:52.566277 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:52.566281 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.566938 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.566939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:52.567134 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:52.567968 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:19:52.567980 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:52.570186 (+ 2206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:52.573523 (+ 3337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:52.573524 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:52.573525 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:52.573878 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:52.574122 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:52.574123 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:52.574123 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:52.574192 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:52.574717 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:52.574718 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:52.574765 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 20:19:52.575457 (+ 692us) client_negotiation.cc:240] Negotiation successful
0928 20:19:52.575712 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":12,"thread_start_us":106,"threads_started":1}
I20260928 20:19:52.576161 10362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:52.560690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.60:38323 (local address 127.3.18.2:33417)
0928 20:19:52.561224 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:52.561234 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:52.562215 (+ 981us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:52.563440 (+ 1225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:52.563441 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:52.563710 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:52.563813 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:52.563815 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.564862 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.564863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:52.566969 (+ 2106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:52.566971 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.567541 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.567541 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:52.567619 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:52.568083 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:19:52.568088 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:52.571283 (+ 3195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:52.573523 (+ 2240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:52.573524 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:52.573525 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:52.574192 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:52.574850 (+ 658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:52.574851 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:52.574852 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:52.574913 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:52.575248 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:52.575248 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:52.575327 (+ 79us) client_negotiation.cc:769] Sending connection context
0928 20:19:52.575855 (+ 528us) client_negotiation.cc:240] Negotiation successful
0928 20:19:52.576001 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"mutex_wait_us":393,"thread_start_us":75,"threads_started":1}
I20260928 20:19:52.576658 10356 heartbeater.cc:347] Connected to a master server at 127.3.18.60:38323
I20260928 20:19:52.576719 10356 heartbeater.cc:464] Registering TS with master...
I20260928 20:19:52.576818 10356 heartbeater.cc:511] Master 127.3.18.60:38323 requested a full tablet report, sending...
I20260928 20:19:52.577308 10040 ts_manager.cc:194] Registered new tserver with Master: 14a5194f37a543ad9cf9c8130a5d11af (127.3.18.2:35465)
I20260928 20:19:52.578030 10358 heartbeater.cc:511] Master 127.3.18.61:33799 requested a full tablet report, sending...
I20260928 20:19:52.578094 10357 heartbeater.cc:347] Connected to a master server at 127.3.18.62:42379
I20260928 20:19:52.578148 10357 heartbeater.cc:464] Registering TS with master...
I20260928 20:19:52.578229 10357 heartbeater.cc:511] Master 127.3.18.62:42379 requested a full tablet report, sending...
I20260928 20:19:52.578397 9854 ts_manager.cc:194] Registered new tserver with Master: 14a5194f37a543ad9cf9c8130a5d11af (127.3.18.2:35465)
I20260928 20:19:52.578474 9257 ts_manager.cc:194] Re-registered known tserver with Master: 14a5194f37a543ad9cf9c8130a5d11af (127.3.18.2:35465)
I20260928 20:19:52.579097 9257 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.18.2@KRBTEST.COM'} at 127.3.18.2:57149
W20260928 20:19:52.658006 10363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:19:52.658236 10363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:19:52.658260 10363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:19:52.659371 10363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:19:52.659409 10363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:19:52.659436 10363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.3
I20260928 20:19:52.660981 10363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:45321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/krb5kdc/oryx_127.3.18.3.keytab
--principal=oryx/127.3.18.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.3:38189
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.18.3
--webserver_port=44589
--tserver_master_addrs=127.3.18.62:42379,127.3.18.61:33799,127.3.18.60:38323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.3
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:19:52.661197 10363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:19:52.661424 10363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:19:52.666937 10369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:52.667142 10371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:52.667155 10368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:52.667353 10363 server_base.cc:1034] running on GCE node
I20260928 20:19:52.667546 10363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:19:52.667757 10363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:19:52.669023 10363 hybrid_clock.cc:648] HybridClock initialized: now 1790626792668990 us; error 43 us; skew 500 ppm
Sep 28 20:19:52 dist-test-slave-b83v krb5kdc[9212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626792, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.18.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:19:52.671849 10363 init.cc:377] Logged in from keytab as oryx/127.3.18.3@KRBTEST.COM (short username oryx)
I20260928 20:19:52.672334 10363 webserver.cc:460] Webserver started at http://127.3.18.3:44589/ using document root <none> and password file <none>
I20260928 20:19:52.672613 10363 fs_manager.cc:359] Metadata directory not provided
I20260928 20:19:52.672701 10363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:19:52.674160 10363 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 20:19:52.674867 10378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:52.675137 10363 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:19:52.675253 10363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-2/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-2/wal
uuid: "1f0e35e0a8fd4cec9d631c0701e0b76f"
format_stamp: "Formatted at 2026-09-28 20:19:51 on dist-test-slave-b83v"
server_key: "f5c1123f66a95cd87d7d92578aac3bb0"
server_key_iv: "9b650ad1c1697cecb65b366777c4c0b0"
server_key_version: "encryptionkey@0"
I20260928 20:19:52.675644 10363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:19:52.686156 10363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:19:52.688442 10363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:19:52.688617 10363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:19:52.688863 10363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:19:52.689226 10363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:19:52.689281 10363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:52.689322 10363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:19:52.689352 10363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:52.695250 10363 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.3:38189
I20260928 20:19:52.695297 10491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.3:38189 every 8 connection(s)
I20260928 20:19:52.695690 10363 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
I20260928 20:19:52.696745 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 10363
Sep 28 20:19:52 dist-test-slave-b83v krb5kdc[9212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626792, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.18.3@KRBTEST.COM for oryx/127.3.18.60@KRBTEST.COM
Sep 28 20:19:52 dist-test-slave-b83v krb5kdc[9212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626792, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.18.3@KRBTEST.COM for oryx/127.3.18.61@KRBTEST.COM
Sep 28 20:19:52 dist-test-slave-b83v krb5kdc[9212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626792, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.18.3@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM
I20260928 20:19:52.710383 9315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:52.696826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:52079 (local address 127.3.18.62:42379)
0928 20:19:52.696879 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:52.696881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:52.697129 (+ 248us) server_negotiation.cc:407] Connection header received
0928 20:19:52.697907 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:52.697909 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:52.697924 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:52.697975 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:52.699104 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.699672 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.701446 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.701794 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.701858 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:52.706410 (+ 4552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:52.706412 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:52.706412 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:52.706421 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:52.708571 (+ 2150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:52.709172 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:52.709173 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:52.709174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:52.709239 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:52.709560 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:52.709562 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:52.709562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:52.709714 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:52.709793 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:52.710202 (+ 409us) server_negotiation.cc:299] Negotiation successful
0928 20:19:52.710285 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 20:19:52.710736 10497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:52.696526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:42379 (local address 127.3.18.3:52079)
0928 20:19:52.697005 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:52.697015 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:52.697766 (+ 751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:52.698138 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:52.698142 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:52.698727 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:52.698978 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:52.698987 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.699840 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.699842 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:52.701305 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:52.701308 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.701915 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.701916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:52.701991 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:52.703043 (+ 1052us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:19:52.703049 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:52.706302 (+ 3253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:52.708674 (+ 2372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:52.708682 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:52.708684 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:52.709069 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:52.709320 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:52.709321 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:52.709321 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:52.709473 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:52.709809 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:52.709818 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:52.710026 (+ 208us) client_negotiation.cc:769] Sending connection context
0928 20:19:52.710169 (+ 143us) client_negotiation.cc:240] Negotiation successful
0928 20:19:52.710370 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"mutex_wait_us":86,"thread_start_us":102,"threads_started":1}
I20260928 20:19:52.711440 10493 heartbeater.cc:347] Connected to a master server at 127.3.18.62:42379
I20260928 20:19:52.711565 10493 heartbeater.cc:464] Registering TS with master...
I20260928 20:19:52.711853 10493 heartbeater.cc:511] Master 127.3.18.62:42379 requested a full tablet report, sending...
I20260928 20:19:52.712203 10225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:52.696827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:40373 (local address 127.3.18.60:38323)
0928 20:19:52.696875 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:52.696878 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:52.697315 (+ 437us) server_negotiation.cc:407] Connection header received
0928 20:19:52.698138 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:52.698139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:52.698152 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:52.698265 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:52.699102 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.699901 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.701412 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.701714 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.701770 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:52.706403 (+ 4633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:52.706404 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:52.706404 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:52.706413 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:52.710132 (+ 3719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:52.710704 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:52.710705 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:52.710705 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:52.710771 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:52.711004 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:52.711005 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:52.711005 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:52.711156 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:52.711234 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:52.711970 (+ 736us) server_negotiation.cc:299] Negotiation successful
0928 20:19:52.712060 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 20:19:52.712293 9257 ts_manager.cc:194] Re-registered known tserver with Master: 1f0e35e0a8fd4cec9d631c0701e0b76f (127.3.18.3:38189)
I20260928 20:19:52.712527 10496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:52.696526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.60:38323 (local address 127.3.18.3:40373)
0928 20:19:52.697232 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:52.697242 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:52.698054 (+ 812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:52.698303 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:52.698304 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:52.698688 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:52.698978 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:52.698985 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.699993 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.699994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:52.701304 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:52.701306 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.701839 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.701839 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:52.701983 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:52.702811 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:19:52.702821 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:52.706301 (+ 3480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:52.710378 (+ 4077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:52.710378 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:52.710379 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:52.710615 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:52.710851 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:52.710852 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:52.710853 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:52.710925 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:52.711243 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:52.711244 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:52.711549 (+ 305us) client_negotiation.cc:769] Sending connection context
0928 20:19:52.712286 (+ 737us) client_negotiation.cc:240] Negotiation successful
0928 20:19:52.712403 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"mutex_wait_us":82,"spinlock_wait_cycles":7936,"thread_start_us":109,"threads_started":1}
I20260928 20:19:52.712858 9257 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.18.3@KRBTEST.COM'} at 127.3.18.3:52079
I20260928 20:19:52.712882 10492 heartbeater.cc:347] Connected to a master server at 127.3.18.60:38323
I20260928 20:19:52.712929 10492 heartbeater.cc:464] Registering TS with master...
I20260928 20:19:52.713006 10492 heartbeater.cc:511] Master 127.3.18.60:38323 requested a full tablet report, sending...
I20260928 20:19:52.713586 10040 ts_manager.cc:194] Registered new tserver with Master: 1f0e35e0a8fd4cec9d631c0701e0b76f (127.3.18.3:38189)
I20260928 20:19:52.714133 10498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:52.696525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.61:33799 (local address 127.3.18.3:51859)
0928 20:19:52.696958 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:52.697010 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:52.697920 (+ 910us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:52.698552 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:52.698552 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:52.698725 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:52.698978 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:52.698985 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.699993 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.699994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:52.701303 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:52.701306 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.701837 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.701839 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:52.701982 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:52.702884 (+ 902us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:19:52.702889 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:52.710286 (+ 7397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:52.712544 (+ 2258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:52.712545 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:52.712546 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:52.713146 (+ 600us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:52.713378 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:52.713378 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:52.713379 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:52.713453 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:52.713821 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:52.713822 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:52.713883 (+ 61us) client_negotiation.cc:769] Sending connection context
0928 20:19:52.713950 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 20:19:52.714044 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"mutex_wait_us":79,"thread_start_us":98,"threads_started":1}
I20260928 20:19:52.714200 9905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:52.696515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:51859 (local address 127.3.18.61:33799)
0928 20:19:52.696780 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:52.696782 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:52.697154 (+ 372us) server_negotiation.cc:407] Connection header received
0928 20:19:52.698019 (+ 865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:52.698021 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:52.698032 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:52.698087 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:52.699101 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.699897 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.701412 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.701711 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.701770 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:52.710364 (+ 8594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:52.710405 (+ 41us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:52.710406 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:52.710415 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:52.712395 (+ 1980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:52.713227 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:52.713228 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:52.713228 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:52.713292 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:52.713529 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:52.713530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:52.713530 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:52.713665 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:52.713731 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:52.714004 (+ 273us) server_negotiation.cc:299] Negotiation successful
0928 20:19:52.714091 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235}
I20260928 20:19:52.714557 10494 heartbeater.cc:347] Connected to a master server at 127.3.18.61:33799
I20260928 20:19:52.714613 10494 heartbeater.cc:464] Registering TS with master...
I20260928 20:19:52.714699 10494 heartbeater.cc:511] Master 127.3.18.61:33799 requested a full tablet report, sending...
I20260928 20:19:52.715078 9854 ts_manager.cc:194] Registered new tserver with Master: 1f0e35e0a8fd4cec9d631c0701e0b76f (127.3.18.3:38189)
I20260928 20:19:52.715452 10504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:52.697788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35270 (local address 127.3.18.62:42379)
0928 20:19:52.703282 (+ 5494us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:52.703284 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:52.703292 (+ 8us) server_negotiation.cc:407] Connection header received
0928 20:19:52.703318 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:52.703319 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:52.703334 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:52.703405 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:52.704266 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.705289 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.706505 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.706799 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.706886 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:52.708272 (+ 1386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:52.708274 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:52.708274 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:52.708281 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:52.710128 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:52.713307 (+ 3179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:52.713309 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:52.713309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:52.713391 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:52.714393 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:52.714394 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:52.714395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:52.714529 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:52.714639 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:52.715076 (+ 437us) server_negotiation.cc:299] Negotiation successful
0928 20:19:52.715173 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5455,"thread_start_us":89,"threads_started":1}
I20260928 20:19:52.716065 10507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:52.698744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60338 (local address 127.3.18.60:38323)
0928 20:19:52.699021 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:52.699023 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:52.704445 (+ 5422us) server_negotiation.cc:407] Connection header received
0928 20:19:52.704477 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:52.704477 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:52.704491 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:52.704552 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:52.705400 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.706240 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.707418 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.707756 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.707879 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:52.709320 (+ 1441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:52.709321 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:52.709321 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:52.709329 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:52.714277 (+ 4948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:52.714645 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:52.714646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:52.714647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:52.714706 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:52.715011 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:52.715013 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:52.715013 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:52.715169 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:52.715735 (+ 566us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:52.715858 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 20:19:52.715966 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":69,"threads_started":1}
I20260928 20:19:52.717978 10506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:52.698507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57478 (local address 127.3.18.61:33799)
0928 20:19:52.699079 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:52.699081 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:52.700208 (+ 1127us) server_negotiation.cc:407] Connection header received
0928 20:19:52.700234 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:52.700235 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:52.700248 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:52.700308 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:52.701165 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.702009 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.703321 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.704085 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.704151 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:52.711344 (+ 7193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:52.711346 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:52.711346 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:52.711356 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:52.716483 (+ 5127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:52.717017 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:52.717019 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:52.717019 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:52.717087 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:52.717318 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:52.717319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:52.717319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:52.717456 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:52.717572 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:52.717680 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 20:19:52.717774 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":58,"threads_started":1}
I20260928 20:19:52.717288 9257 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35270:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 20:19:52.719050 9257 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 20:19:52.727686 10513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:52.724156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.2:35465 (local address 127.0.0.1:43402)
0928 20:19:52.724578 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:52.724588 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:52.724639 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:52.724818 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:52.724819 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:52.724821 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:19:52.724874 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:52.724876 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.725847 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.725848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:52.726681 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:52.726684 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.727431 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.727432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:52.727503 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:52.727509 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 20:19:52.727554 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 20:19:52.727601 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":82,"threads_started":1}
I20260928 20:19:52.727694 10516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:52.724262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43402 (local address 127.3.18.2:35465)
0928 20:19:52.724441 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:52.724445 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:52.724640 (+ 195us) server_negotiation.cc:407] Connection header received
0928 20:19:52.724703 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:52.724705 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:52.724721 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:52.724824 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:19:52.725033 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.725747 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.726786 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.727335 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.727431 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:52.727528 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:52.727568 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 20:19:52.727607 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":88,"threads_started":1}
I20260928 20:19:52.728644 9814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:52.723832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.1:44945 (local address 127.0.0.1:59926)
0928 20:19:52.723884 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:52.723897 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:52.723933 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:52.724701 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:52.724702 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:52.724704 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:19:52.724761 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:52.724763 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.726263 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.726264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:52.727370 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:52.727373 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.728386 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.728388 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:52.728452 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:52.728455 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 20:19:52.728490 (+ 35us) client_negotiation.cc:240] Negotiation successful
0928 20:19:52.728544 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":2}
I20260928 20:19:52.728811 10514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:52.724080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53470 (local address 127.3.18.3:38189)
0928 20:19:52.724309 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:52.724314 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:52.724522 (+ 208us) server_negotiation.cc:407] Connection header received
0928 20:19:52.724621 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:52.724623 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:52.724637 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:52.724712 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:19:52.724911 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.726053 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.727497 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.728281 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.728383 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:52.728473 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:52.728680 (+ 207us) server_negotiation.cc:299] Negotiation successful
0928 20:19:52.728731 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":126,"threads_started":1}
I20260928 20:19:52.729040 10290 tablet_service.cc:1467] Processing CreateTablet for tablet fd5e92ec6f894b70a9b2c8fe743159d9 (DEFAULT_TABLE table=test-table [id=59889f5869174064861621e97aa2abe3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:19:52.729492 10290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd5e92ec6f894b70a9b2c8fe743159d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:52.729504 10515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:52.723918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.3:38189 (local address 127.0.0.1:53470)
0928 20:19:52.724453 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:52.724465 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:52.724533 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:52.724789 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:52.724790 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:52.724792 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:19:52.724829 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:52.724830 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.726161 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.726162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:52.727393 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:52.727395 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.728534 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.728534 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:52.728590 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:52.728593 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 20:19:52.729252 (+ 659us) client_negotiation.cc:240] Negotiation successful
0928 20:19:52.729325 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"mutex_wait_us":3,"thread_start_us":118,"threads_started":1}
I20260928 20:19:52.729636 10517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:52.724326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59926 (local address 127.3.18.1:44945)
0928 20:19:52.724503 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:52.724507 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:52.724518 (+ 11us) server_negotiation.cc:407] Connection header received
0928 20:19:52.724563 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:52.724565 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:52.724585 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:52.724837 (+ 252us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:19:52.724914 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.726177 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.727478 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.728281 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.728770 (+ 489us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:52.729462 (+ 692us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:52.729498 (+ 36us) server_negotiation.cc:299] Negotiation successful
0928 20:19:52.729557 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":87,"threads_started":1}
I20260928 20:19:52.730363 10153 tablet_service.cc:1467] Processing CreateTablet for tablet fd5e92ec6f894b70a9b2c8fe743159d9 (DEFAULT_TABLE table=test-table [id=59889f5869174064861621e97aa2abe3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:19:52.730363 10426 tablet_service.cc:1467] Processing CreateTablet for tablet fd5e92ec6f894b70a9b2c8fe743159d9 (DEFAULT_TABLE table=test-table [id=59889f5869174064861621e97aa2abe3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:19:52.730720 10426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd5e92ec6f894b70a9b2c8fe743159d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:52.730720 10153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd5e92ec6f894b70a9b2c8fe743159d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:52.732903 10518 tablet_bootstrap.cc:492] T fd5e92ec6f894b70a9b2c8fe743159d9 P 14a5194f37a543ad9cf9c8130a5d11af: Bootstrap starting.
I20260928 20:19:52.733611 10518 tablet_bootstrap.cc:654] T fd5e92ec6f894b70a9b2c8fe743159d9 P 14a5194f37a543ad9cf9c8130a5d11af: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:52.734002 10518 log.cc:824] T fd5e92ec6f894b70a9b2c8fe743159d9 P 14a5194f37a543ad9cf9c8130a5d11af: Log is configured to *not* fsync() on all Append() calls
I20260928 20:19:52.734889 10518 tablet_bootstrap.cc:492] T fd5e92ec6f894b70a9b2c8fe743159d9 P 14a5194f37a543ad9cf9c8130a5d11af: No bootstrap required, opened a new log
I20260928 20:19:52.734975 10518 ts_tablet_manager.cc:1397] T fd5e92ec6f894b70a9b2c8fe743159d9 P 14a5194f37a543ad9cf9c8130a5d11af: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 20:19:52.734951 10520 tablet_bootstrap.cc:492] T fd5e92ec6f894b70a9b2c8fe743159d9 P 1f0e35e0a8fd4cec9d631c0701e0b76f: Bootstrap starting.
I20260928 20:19:52.735707 10520 tablet_bootstrap.cc:654] T fd5e92ec6f894b70a9b2c8fe743159d9 P 1f0e35e0a8fd4cec9d631c0701e0b76f: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:52.736042 10521 tablet_bootstrap.cc:492] T fd5e92ec6f894b70a9b2c8fe743159d9 P 39e67e2773bb4225a13ec3d8425a088a: Bootstrap starting.
I20260928 20:19:52.736122 10520 log.cc:824] T fd5e92ec6f894b70a9b2c8fe743159d9 P 1f0e35e0a8fd4cec9d631c0701e0b76f: Log is configured to *not* fsync() on all Append() calls
I20260928 20:19:52.736824 10521 tablet_bootstrap.cc:654] T fd5e92ec6f894b70a9b2c8fe743159d9 P 39e67e2773bb4225a13ec3d8425a088a: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:52.737135 10521 log.cc:824] T fd5e92ec6f894b70a9b2c8fe743159d9 P 39e67e2773bb4225a13ec3d8425a088a: Log is configured to *not* fsync() on all Append() calls
I20260928 20:19:52.737156 10520 tablet_bootstrap.cc:492] T fd5e92ec6f894b70a9b2c8fe743159d9 P 1f0e35e0a8fd4cec9d631c0701e0b76f: No bootstrap required, opened a new log
I20260928 20:19:52.737229 10520 ts_tablet_manager.cc:1397] T fd5e92ec6f894b70a9b2c8fe743159d9 P 1f0e35e0a8fd4cec9d631c0701e0b76f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 20:19:52.737855 10521 tablet_bootstrap.cc:492] T fd5e92ec6f894b70a9b2c8fe743159d9 P 39e67e2773bb4225a13ec3d8425a088a: No bootstrap required, opened a new log
I20260928 20:19:52.737973 10521 ts_tablet_manager.cc:1397] T fd5e92ec6f894b70a9b2c8fe743159d9 P 39e67e2773bb4225a13ec3d8425a088a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 20:19:52.738272 10518 raft_consensus.cc:348] T fd5e92ec6f894b70a9b2c8fe743159d9 P 14a5194f37a543ad9cf9c8130a5d11af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39e67e2773bb4225a13ec3d8425a088a" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 44945 } } peers { permanent_uuid: "1f0e35e0a8fd4cec9d631c0701e0b76f" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38189 } } peers { permanent_uuid: "14a5194f37a543ad9cf9c8130a5d11af" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 35465 } }
I20260928 20:19:52.738552 10518 raft_consensus.cc:374] T fd5e92ec6f894b70a9b2c8fe743159d9 P 14a5194f37a543ad9cf9c8130a5d11af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:52.738807 10518 raft_consensus.cc:729] T fd5e92ec6f894b70a9b2c8fe743159d9 P 14a5194f37a543ad9cf9c8130a5d11af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14a5194f37a543ad9cf9c8130a5d11af, State: Initialized, Role: FOLLOWER
I20260928 20:19:52.738981 10518 consensus_queue.cc:260] T fd5e92ec6f894b70a9b2c8fe743159d9 P 14a5194f37a543ad9cf9c8130a5d11af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39e67e2773bb4225a13ec3d8425a088a" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 44945 } } peers { permanent_uuid: "1f0e35e0a8fd4cec9d631c0701e0b76f" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38189 } } peers { permanent_uuid: "14a5194f37a543ad9cf9c8130a5d11af" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 35465 } }
I20260928 20:19:52.739317 10518 ts_tablet_manager.cc:1428] T fd5e92ec6f894b70a9b2c8fe743159d9 P 14a5194f37a543ad9cf9c8130a5d11af: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260928 20:19:52.739533 10357 heartbeater.cc:503] Master 127.3.18.62:42379 was elected leader, sending a full tablet report...
I20260928 20:19:52.740345 10521 raft_consensus.cc:348] T fd5e92ec6f894b70a9b2c8fe743159d9 P 39e67e2773bb4225a13ec3d8425a088a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39e67e2773bb4225a13ec3d8425a088a" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 44945 } } peers { permanent_uuid: "1f0e35e0a8fd4cec9d631c0701e0b76f" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38189 } } peers { permanent_uuid: "14a5194f37a543ad9cf9c8130a5d11af" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 35465 } }
I20260928 20:19:52.740558 10521 raft_consensus.cc:374] T fd5e92ec6f894b70a9b2c8fe743159d9 P 39e67e2773bb4225a13ec3d8425a088a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:52.740604 10521 raft_consensus.cc:729] T fd5e92ec6f894b70a9b2c8fe743159d9 P 39e67e2773bb4225a13ec3d8425a088a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39e67e2773bb4225a13ec3d8425a088a, State: Initialized, Role: FOLLOWER
I20260928 20:19:52.740731 10521 consensus_queue.cc:260] T fd5e92ec6f894b70a9b2c8fe743159d9 P 39e67e2773bb4225a13ec3d8425a088a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39e67e2773bb4225a13ec3d8425a088a" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 44945 } } peers { permanent_uuid: "1f0e35e0a8fd4cec9d631c0701e0b76f" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38189 } } peers { permanent_uuid: "14a5194f37a543ad9cf9c8130a5d11af" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 35465 } }
I20260928 20:19:52.741016 10521 ts_tablet_manager.cc:1428] T fd5e92ec6f894b70a9b2c8fe743159d9 P 39e67e2773bb4225a13ec3d8425a088a: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 20:19:52.740842 10520 raft_consensus.cc:348] T fd5e92ec6f894b70a9b2c8fe743159d9 P 1f0e35e0a8fd4cec9d631c0701e0b76f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39e67e2773bb4225a13ec3d8425a088a" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 44945 } } peers { permanent_uuid: "1f0e35e0a8fd4cec9d631c0701e0b76f" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38189 } } peers { permanent_uuid: "14a5194f37a543ad9cf9c8130a5d11af" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 35465 } }
I20260928 20:19:52.741072 10520 raft_consensus.cc:374] T fd5e92ec6f894b70a9b2c8fe743159d9 P 1f0e35e0a8fd4cec9d631c0701e0b76f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:52.741117 10520 raft_consensus.cc:729] T fd5e92ec6f894b70a9b2c8fe743159d9 P 1f0e35e0a8fd4cec9d631c0701e0b76f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0e35e0a8fd4cec9d631c0701e0b76f, State: Initialized, Role: FOLLOWER
I20260928 20:19:52.741250 10520 consensus_queue.cc:260] T fd5e92ec6f894b70a9b2c8fe743159d9 P 1f0e35e0a8fd4cec9d631c0701e0b76f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39e67e2773bb4225a13ec3d8425a088a" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 44945 } } peers { permanent_uuid: "1f0e35e0a8fd4cec9d631c0701e0b76f" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38189 } } peers { permanent_uuid: "14a5194f37a543ad9cf9c8130a5d11af" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 35465 } }
I20260928 20:19:52.741531 10520 ts_tablet_manager.cc:1428] T fd5e92ec6f894b70a9b2c8fe743159d9 P 1f0e35e0a8fd4cec9d631c0701e0b76f: Time spent starting tablet: real 0.004s user 0.003s sys 0.002s
I20260928 20:19:52.741562 10220 heartbeater.cc:503] Master 127.3.18.62:42379 was elected leader, sending a full tablet report...
I20260928 20:19:52.742069 10493 heartbeater.cc:503] Master 127.3.18.62:42379 was elected leader, sending a full tablet report...
I20260928 20:19:52.783174 10515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:52.780239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.60:38323 (local address 127.0.0.1:60348)
0928 20:19:52.780313 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:52.780325 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:52.780401 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:52.780605 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:52.780606 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:52.780608 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:19:52.780669 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:52.780671 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.781413 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.781415 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:52.782256 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:52.782259 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.782842 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.782843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:52.782909 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:52.782913 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:19:52.783042 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 20:19:52.783095 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 20:19:52.783174 10507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:52.780319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60348 (local address 127.3.18.60:38323)
0928 20:19:52.780452 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:52.780455 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:52.780463 (+ 8us) server_negotiation.cc:407] Connection header received
0928 20:19:52.780502 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:52.780503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:52.780523 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:52.780599 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:19:52.780738 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.781318 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.782329 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.782742 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.782839 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:52.782930 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:52.783050 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 20:19:52.783095 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260928 20:19:52.783835 10050 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 645a6ba8e676428585dc367ef37372c4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260928 20:19:52.784880 10079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 645a6ba8e676428585dc367ef37372c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef4554cccf754a53a7d74f685421731a. Latest consensus state: current_term: 1 leader_uuid: "ef4554cccf754a53a7d74f685421731a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ef4554cccf754a53a7d74f685421731a" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 42379 } } peers { permanent_uuid: "d13b62b954784d05bd09150e14f708b7" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 33799 } attrs { promote: false } } peers { permanent_uuid: "645a6ba8e676428585dc367ef37372c4" member_type: NON_VOTER last_known_addr { host: "127.3.18.60" port: 38323 } attrs { promote: true } } }
I20260928 20:19:52.785029 10079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 645a6ba8e676428585dc367ef37372c4 [sys.catalog]: This master's current role is: LEARNER
W20260928 20:19:52.811511 10359 tablet.cc:2369] T fd5e92ec6f894b70a9b2c8fe743159d9 P 14a5194f37a543ad9cf9c8130a5d11af: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 20:19:52.910068 10222 tablet.cc:2369] T fd5e92ec6f894b70a9b2c8fe743159d9 P 39e67e2773bb4225a13ec3d8425a088a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 20:19:52.947623 10495 tablet.cc:2369] T fd5e92ec6f894b70a9b2c8fe743159d9 P 1f0e35e0a8fd4cec9d631c0701e0b76f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 20:19:52.948660 10526 raft_consensus.cc:482] T fd5e92ec6f894b70a9b2c8fe743159d9 P 1f0e35e0a8fd4cec9d631c0701e0b76f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:19:52.948755 10526 raft_consensus.cc:504] T fd5e92ec6f894b70a9b2c8fe743159d9 P 1f0e35e0a8fd4cec9d631c0701e0b76f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39e67e2773bb4225a13ec3d8425a088a" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 44945 } } peers { permanent_uuid: "1f0e35e0a8fd4cec9d631c0701e0b76f" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38189 } } peers { permanent_uuid: "14a5194f37a543ad9cf9c8130a5d11af" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 35465 } }
I20260928 20:19:52.949237 10526 leader_election.cc:288] T fd5e92ec6f894b70a9b2c8fe743159d9 P 1f0e35e0a8fd4cec9d631c0701e0b76f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39e67e2773bb4225a13ec3d8425a088a (127.3.18.1:44945), 14a5194f37a543ad9cf9c8130a5d11af (127.3.18.2:35465)
I20260928 20:19:52.952306 10516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:52.949481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:60165 (local address 127.3.18.2:35465)
0928 20:19:52.949652 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:52.949654 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:52.949661 (+ 7us) server_negotiation.cc:407] Connection header received
0928 20:19:52.949687 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:52.949692 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:52.949703 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:52.949771 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:19:52.949966 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.950429 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.951533 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.951944 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.952021 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:52.952032 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:52.952188 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 20:19:52.952232 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149}
I20260928 20:19:52.952320 10498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:52.949449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.2:35465 (local address 127.3.18.3:60165)
0928 20:19:52.949539 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:52.949550 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:52.949594 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:52.949793 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:52.949794 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:52.949796 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:19:52.949857 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:52.949858 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.950594 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.950595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:52.951408 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:52.951410 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.952033 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.952035 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:52.952109 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:52.952111 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 20:19:52.952182 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 20:19:52.952229 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72,"mutex_wait_us":7}
I20260928 20:19:52.952304 10517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:52.949587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:43151 (local address 127.3.18.1:44945)
0928 20:19:52.949633 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:52.949635 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:52.949644 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:19:52.949673 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:52.949674 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:52.949686 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:52.949741 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:19:52.949955 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.950458 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.951500 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.951938 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.952022 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:52.952033 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:52.952191 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 20:19:52.952229 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 20:19:52.952343 10497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:52.949434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.1:44945 (local address 127.3.18.3:43151)
0928 20:19:52.949529 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:52.949542 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:52.949589 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:52.949765 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:52.949766 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:52.949768 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:19:52.949840 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:52.949842 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.950575 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.950576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:52.951399 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:52.951401 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:52.952023 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:52.952024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:52.952098 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:52.952101 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 20:19:52.952173 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 20:19:52.952214 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71,"mutex_wait_us":30}
I20260928 20:19:52.952741 10310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd5e92ec6f894b70a9b2c8fe743159d9" candidate_uuid: "1f0e35e0a8fd4cec9d631c0701e0b76f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14a5194f37a543ad9cf9c8130a5d11af" is_pre_election: true
I20260928 20:19:52.952864 10173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd5e92ec6f894b70a9b2c8fe743159d9" candidate_uuid: "1f0e35e0a8fd4cec9d631c0701e0b76f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39e67e2773bb4225a13ec3d8425a088a" is_pre_election: true
I20260928 20:19:52.952950 10310 raft_consensus.cc:2454] T fd5e92ec6f894b70a9b2c8fe743159d9 P 14a5194f37a543ad9cf9c8130a5d11af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f0e35e0a8fd4cec9d631c0701e0b76f in term 0.
I20260928 20:19:52.953030 10173 raft_consensus.cc:2454] T fd5e92ec6f894b70a9b2c8fe743159d9 P 39e67e2773bb4225a13ec3d8425a088a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f0e35e0a8fd4cec9d631c0701e0b76f in term 0.
I20260928 20:19:52.953191 10379 leader_election.cc:302] T fd5e92ec6f894b70a9b2c8fe743159d9 P 1f0e35e0a8fd4cec9d631c0701e0b76f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14a5194f37a543ad9cf9c8130a5d11af, 1f0e35e0a8fd4cec9d631c0701e0b76f; no voters:
I20260928 20:19:52.953321 10526 raft_consensus.cc:2781] T fd5e92ec6f894b70a9b2c8fe743159d9 P 1f0e35e0a8fd4cec9d631c0701e0b76f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 20:19:52.953377 10526 raft_consensus.cc:482] T fd5e92ec6f894b70a9b2c8fe743159d9 P 1f0e35e0a8fd4cec9d631c0701e0b76f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 20:19:52.953405 10526 raft_consensus.cc:3037] T fd5e92ec6f894b70a9b2c8fe743159d9 P 1f0e35e0a8fd4cec9d631c0701e0b76f [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:52.954560 10526 raft_consensus.cc:504] T fd5e92ec6f894b70a9b2c8fe743159d9 P 1f0e35e0a8fd4cec9d631c0701e0b76f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39e67e2773bb4225a13ec3d8425a088a" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 44945 } } peers { permanent_uuid: "1f0e35e0a8fd4cec9d631c0701e0b76f" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38189 } } peers { permanent_uuid: "14a5194f37a543ad9cf9c8130a5d11af" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 35465 } }
I20260928 20:19:52.954754 10526 leader_election.cc:288] T fd5e92ec6f894b70a9b2c8fe743159d9 P 1f0e35e0a8fd4cec9d631c0701e0b76f [CANDIDATE]: Term 1 election: Requested vote from peers 39e67e2773bb4225a13ec3d8425a088a (127.3.18.1:44945), 14a5194f37a543ad9cf9c8130a5d11af (127.3.18.2:35465)
I20260928 20:19:52.954991 10173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd5e92ec6f894b70a9b2c8fe743159d9" candidate_uuid: "1f0e35e0a8fd4cec9d631c0701e0b76f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39e67e2773bb4225a13ec3d8425a088a"
I20260928 20:19:52.954977 10310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd5e92ec6f894b70a9b2c8fe743159d9" candidate_uuid: "1f0e35e0a8fd4cec9d631c0701e0b76f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14a5194f37a543ad9cf9c8130a5d11af"
I20260928 20:19:52.955092 10310 raft_consensus.cc:3037] T fd5e92ec6f894b70a9b2c8fe743159d9 P 14a5194f37a543ad9cf9c8130a5d11af [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:52.955094 10173 raft_consensus.cc:3037] T fd5e92ec6f894b70a9b2c8fe743159d9 P 39e67e2773bb4225a13ec3d8425a088a [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:52.956192 10173 raft_consensus.cc:2454] T fd5e92ec6f894b70a9b2c8fe743159d9 P 39e67e2773bb4225a13ec3d8425a088a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f0e35e0a8fd4cec9d631c0701e0b76f in term 1.
I20260928 20:19:52.956215 10310 raft_consensus.cc:2454] T fd5e92ec6f894b70a9b2c8fe743159d9 P 14a5194f37a543ad9cf9c8130a5d11af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f0e35e0a8fd4cec9d631c0701e0b76f in term 1.
I20260928 20:19:52.956391 10379 leader_election.cc:302] T fd5e92ec6f894b70a9b2c8fe743159d9 P 1f0e35e0a8fd4cec9d631c0701e0b76f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14a5194f37a543ad9cf9c8130a5d11af, 1f0e35e0a8fd4cec9d631c0701e0b76f; no voters:
I20260928 20:19:52.956501 10526 raft_consensus.cc:2781] T fd5e92ec6f894b70a9b2c8fe743159d9 P 1f0e35e0a8fd4cec9d631c0701e0b76f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:19:52.956652 10526 raft_consensus.cc:686] T fd5e92ec6f894b70a9b2c8fe743159d9 P 1f0e35e0a8fd4cec9d631c0701e0b76f [term 1 LEADER]: Becoming Leader. State: Replica: 1f0e35e0a8fd4cec9d631c0701e0b76f, State: Running, Role: LEADER
I20260928 20:19:52.956764 10526 consensus_queue.cc:237] T fd5e92ec6f894b70a9b2c8fe743159d9 P 1f0e35e0a8fd4cec9d631c0701e0b76f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39e67e2773bb4225a13ec3d8425a088a" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 44945 } } peers { permanent_uuid: "1f0e35e0a8fd4cec9d631c0701e0b76f" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38189 } } peers { permanent_uuid: "14a5194f37a543ad9cf9c8130a5d11af" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 35465 } }
I20260928 20:19:52.957769 9257 catalog_manager.cc:5495] T fd5e92ec6f894b70a9b2c8fe743159d9 P 1f0e35e0a8fd4cec9d631c0701e0b76f reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f0e35e0a8fd4cec9d631c0701e0b76f (127.3.18.3). New cstate: current_term: 1 leader_uuid: "1f0e35e0a8fd4cec9d631c0701e0b76f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39e67e2773bb4225a13ec3d8425a088a" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 44945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f0e35e0a8fd4cec9d631c0701e0b76f" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14a5194f37a543ad9cf9c8130a5d11af" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 35465 } health_report { overall_health: UNKNOWN } } }
I20260928 20:19:52.958460 9816 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef4554cccf754a53a7d74f685421731a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ef4554cccf754a53a7d74f685421731a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ef4554cccf754a53a7d74f685421731a" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 42379 } } peers { permanent_uuid: "d13b62b954784d05bd09150e14f708b7" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 33799 } attrs { promote: false } } peers { permanent_uuid: "645a6ba8e676428585dc367ef37372c4" member_type: NON_VOTER last_known_addr { host: "127.3.18.60" port: 38323 } attrs { promote: true } } }
I20260928 20:19:52.958554 9816 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef4554cccf754a53a7d74f685421731a [sys.catalog]: This master's current role is: LEADER
I20260928 20:19:52.960758 10004 raft_consensus.cc:1050] T 00000000000000000000000000000000 P ef4554cccf754a53a7d74f685421731a: attempting to promote NON_VOTER 645a6ba8e676428585dc367ef37372c4 to VOTER
I20260928 20:19:52.960909 10004 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef4554cccf754a53a7d74f685421731a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ef4554cccf754a53a7d74f685421731a" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 42379 } } peers { permanent_uuid: "d13b62b954784d05bd09150e14f708b7" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 33799 } attrs { promote: false } } peers { permanent_uuid: "645a6ba8e676428585dc367ef37372c4" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 38323 } attrs { promote: false } }
I20260928 20:19:52.961548 9864 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d13b62b954784d05bd09150e14f708b7 [term 1 FOLLOWER]: Refusing update from remote peer ef4554cccf754a53a7d74f685421731a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260928 20:19:52.961697 10050 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 645a6ba8e676428585dc367ef37372c4 [term 1 LEARNER]: Refusing update from remote peer ef4554cccf754a53a7d74f685421731a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260928 20:19:52.961799 9816 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ef4554cccf754a53a7d74f685421731a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13b62b954784d05bd09150e14f708b7" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 33799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260928 20:19:52.961917 9816 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ef4554cccf754a53a7d74f685421731a [LEADER]: Connected to new peer: Peer: permanent_uuid: "645a6ba8e676428585dc367ef37372c4" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 38323 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260928 20:19:52.962585 9816 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ef4554cccf754a53a7d74f685421731a [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 645a6ba8e676428585dc367ef37372c4 (127.3.18.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ef4554cccf754a53a7d74f685421731a" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 42379 } } peers { permanent_uuid: "d13b62b954784d05bd09150e14f708b7" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 33799 } attrs { promote: false } } peers { permanent_uuid: "645a6ba8e676428585dc367ef37372c4" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 38323 } attrs { promote: false } } }
I20260928 20:19:52.962762 10050 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 645a6ba8e676428585dc367ef37372c4 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 645a6ba8e676428585dc367ef37372c4 (127.3.18.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ef4554cccf754a53a7d74f685421731a" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 42379 } } peers { permanent_uuid: "d13b62b954784d05bd09150e14f708b7" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 33799 } attrs { promote: false } } peers { permanent_uuid: "645a6ba8e676428585dc367ef37372c4" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 38323 } attrs { promote: false } } }
I20260928 20:19:52.962957 9864 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d13b62b954784d05bd09150e14f708b7 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 645a6ba8e676428585dc367ef37372c4 (127.3.18.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ef4554cccf754a53a7d74f685421731a" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 42379 } } peers { permanent_uuid: "d13b62b954784d05bd09150e14f708b7" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 33799 } attrs { promote: false } } peers { permanent_uuid: "645a6ba8e676428585dc367ef37372c4" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 38323 } attrs { promote: false } } }
I20260928 20:19:52.963593 9816 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef4554cccf754a53a7d74f685421731a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ef4554cccf754a53a7d74f685421731a" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ef4554cccf754a53a7d74f685421731a" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 42379 } } peers { permanent_uuid: "d13b62b954784d05bd09150e14f708b7" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 33799 } attrs { promote: false } } peers { permanent_uuid: "645a6ba8e676428585dc367ef37372c4" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 38323 } attrs { promote: false } } }
I20260928 20:19:52.963675 9816 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef4554cccf754a53a7d74f685421731a [sys.catalog]: This master's current role is: LEADER
I20260928 20:19:52.963747 9816 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef4554cccf754a53a7d74f685421731a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ef4554cccf754a53a7d74f685421731a" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ef4554cccf754a53a7d74f685421731a" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 42379 } } peers { permanent_uuid: "d13b62b954784d05bd09150e14f708b7" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 33799 } attrs { promote: false } } peers { permanent_uuid: "645a6ba8e676428585dc367ef37372c4" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 38323 } attrs { promote: false } } }
I20260928 20:19:52.963829 9815 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef4554cccf754a53a7d74f685421731a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ef4554cccf754a53a7d74f685421731a" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ef4554cccf754a53a7d74f685421731a" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 42379 } } peers { permanent_uuid: "d13b62b954784d05bd09150e14f708b7" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 33799 } attrs { promote: false } } peers { permanent_uuid: "645a6ba8e676428585dc367ef37372c4" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 38323 } attrs { promote: false } } }
I20260928 20:19:52.963915 9815 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef4554cccf754a53a7d74f685421731a [sys.catalog]: This master's current role is: LEADER
I20260928 20:19:52.963902 10079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 645a6ba8e676428585dc367ef37372c4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ef4554cccf754a53a7d74f685421731a" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ef4554cccf754a53a7d74f685421731a" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 42379 } } peers { permanent_uuid: "d13b62b954784d05bd09150e14f708b7" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 33799 } attrs { promote: false } } peers { permanent_uuid: "645a6ba8e676428585dc367ef37372c4" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 38323 } attrs { promote: false } } }
I20260928 20:19:52.964016 10079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 645a6ba8e676428585dc367ef37372c4 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 20:19:52.963836 9816 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef4554cccf754a53a7d74f685421731a [sys.catalog]: This master's current role is: LEADER
I20260928 20:19:52.964241 10534 sys_catalog.cc:455] T 00000000000000000000000000000000 P d13b62b954784d05bd09150e14f708b7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ef4554cccf754a53a7d74f685421731a" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ef4554cccf754a53a7d74f685421731a" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 42379 } } peers { permanent_uuid: "d13b62b954784d05bd09150e14f708b7" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 33799 } attrs { promote: false } } peers { permanent_uuid: "645a6ba8e676428585dc367ef37372c4" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 38323 } attrs { promote: false } } }
I20260928 20:19:52.964319 10534 sys_catalog.cc:458] T 00000000000000000000000000000000 P d13b62b954784d05bd09150e14f708b7 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 20:19:53.011382 10514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:53.008448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53480 (local address 127.3.18.3:38189)
0928 20:19:53.008518 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:53.008520 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:53.008583 (+ 63us) server_negotiation.cc:407] Connection header received
0928 20:19:53.008665 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:53.008666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:53.008688 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:53.008763 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:19:53.009026 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:53.009566 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:53.010389 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:53.010620 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:53.010692 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:53.010963 (+ 271us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:19:53.011079 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:19:53.011150 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:53.011242 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 20:19:53.011263 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 20:19:53.013433 10173 raft_consensus.cc:1261] T fd5e92ec6f894b70a9b2c8fe743159d9 P 39e67e2773bb4225a13ec3d8425a088a [term 1 FOLLOWER]: Refusing update from remote peer 1f0e35e0a8fd4cec9d631c0701e0b76f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:19:53.013451 10310 raft_consensus.cc:1261] T fd5e92ec6f894b70a9b2c8fe743159d9 P 14a5194f37a543ad9cf9c8130a5d11af [term 1 FOLLOWER]: Refusing update from remote peer 1f0e35e0a8fd4cec9d631c0701e0b76f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:19:53.013868 10531 consensus_queue.cc:1035] T fd5e92ec6f894b70a9b2c8fe743159d9 P 1f0e35e0a8fd4cec9d631c0701e0b76f [LEADER]: Connected to new peer: Peer: permanent_uuid: "39e67e2773bb4225a13ec3d8425a088a" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 44945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:19:53.014014 10526 consensus_queue.cc:1035] T fd5e92ec6f894b70a9b2c8fe743159d9 P 1f0e35e0a8fd4cec9d631c0701e0b76f [LEADER]: Connected to new peer: Peer: permanent_uuid: "14a5194f37a543ad9cf9c8130a5d11af" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 35465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:19:53.016470 10537 mvcc.cc:205] Tried to move back new op lower bound from 7334407344179720192 to 7334407343951429632. Current Snapshot: MvccSnapshot[applied={T|T < 7334407344179720192}]
I20260928 20:19:53.017558 10536 mvcc.cc:205] Tried to move back new op lower bound from 7334407344179720192 to 7334407343951429632. Current Snapshot: MvccSnapshot[applied={T|T < 7334407344179720192 or (T in {7334407344179720192})}]
I20260928 20:19:53.018980 9256 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35270:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:19:53.019078 9256 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35270:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:19:53.020511 9256 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ef4554cccf754a53a7d74f685421731a: Sending DeleteTablet for 3 replicas of tablet fd5e92ec6f894b70a9b2c8fe743159d9
I20260928 20:19:53.020859 10153 tablet_service.cc:1514] Processing DeleteTablet for tablet fd5e92ec6f894b70a9b2c8fe743159d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:19:53 UTC) from {username='oryx', principal='oryx/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:59926
I20260928 20:19:53.020862 10426 tablet_service.cc:1514] Processing DeleteTablet for tablet fd5e92ec6f894b70a9b2c8fe743159d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:19:53 UTC) from {username='oryx', principal='oryx/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:53470
I20260928 20:19:53.020869 10290 tablet_service.cc:1514] Processing DeleteTablet for tablet fd5e92ec6f894b70a9b2c8fe743159d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:19:53 UTC) from {username='oryx', principal='oryx/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:43402
I20260928 20:19:53.021086 10548 tablet_replica.cc:330] T fd5e92ec6f894b70a9b2c8fe743159d9 P 1f0e35e0a8fd4cec9d631c0701e0b76f: stopping tablet replica
I20260928 20:19:53.021184 10548 raft_consensus.cc:2229] T fd5e92ec6f894b70a9b2c8fe743159d9 P 1f0e35e0a8fd4cec9d631c0701e0b76f [term 1 LEADER]: Raft consensus shutting down.
I20260928 20:19:53.021281 10548 raft_consensus.cc:2258] T fd5e92ec6f894b70a9b2c8fe743159d9 P 1f0e35e0a8fd4cec9d631c0701e0b76f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:19:53.021592 10546 tablet_replica.cc:330] T fd5e92ec6f894b70a9b2c8fe743159d9 P 39e67e2773bb4225a13ec3d8425a088a: stopping tablet replica
I20260928 20:19:53.021692 10546 raft_consensus.cc:2229] T fd5e92ec6f894b70a9b2c8fe743159d9 P 39e67e2773bb4225a13ec3d8425a088a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:19:53.021803 10546 raft_consensus.cc:2258] T fd5e92ec6f894b70a9b2c8fe743159d9 P 39e67e2773bb4225a13ec3d8425a088a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:19:53.022116 10548 ts_tablet_manager.cc:1905] T fd5e92ec6f894b70a9b2c8fe743159d9 P 1f0e35e0a8fd4cec9d631c0701e0b76f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:19:53.022238 10546 ts_tablet_manager.cc:1905] T fd5e92ec6f894b70a9b2c8fe743159d9 P 39e67e2773bb4225a13ec3d8425a088a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:19:53.022400 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 9225
I20260928 20:19:53.023871 10547 tablet_replica.cc:330] T fd5e92ec6f894b70a9b2c8fe743159d9 P 14a5194f37a543ad9cf9c8130a5d11af: stopping tablet replica
I20260928 20:19:53.023948 10547 raft_consensus.cc:2229] T fd5e92ec6f894b70a9b2c8fe743159d9 P 14a5194f37a543ad9cf9c8130a5d11af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:19:53.024017 10547 raft_consensus.cc:2258] T fd5e92ec6f894b70a9b2c8fe743159d9 P 14a5194f37a543ad9cf9c8130a5d11af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:19:53.024178 10548 ts_tablet_manager.cc:1918] T fd5e92ec6f894b70a9b2c8fe743159d9 P 1f0e35e0a8fd4cec9d631c0701e0b76f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:19:53.024241 10548 log.cc:1196] T fd5e92ec6f894b70a9b2c8fe743159d9 P 1f0e35e0a8fd4cec9d631c0701e0b76f: Deleting WAL directory at /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-2/wal/wals/fd5e92ec6f894b70a9b2c8fe743159d9
I20260928 20:19:53.024489 10547 ts_tablet_manager.cc:1905] T fd5e92ec6f894b70a9b2c8fe743159d9 P 14a5194f37a543ad9cf9c8130a5d11af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:19:53.024525 10548 ts_tablet_manager.cc:1939] T fd5e92ec6f894b70a9b2c8fe743159d9 P 1f0e35e0a8fd4cec9d631c0701e0b76f: Deleting consensus metadata
I20260928 20:19:53.024881 10546 ts_tablet_manager.cc:1918] T fd5e92ec6f894b70a9b2c8fe743159d9 P 39e67e2773bb4225a13ec3d8425a088a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:19:53.024950 10546 log.cc:1196] T fd5e92ec6f894b70a9b2c8fe743159d9 P 39e67e2773bb4225a13ec3d8425a088a: Deleting WAL directory at /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-0/wal/wals/fd5e92ec6f894b70a9b2c8fe743159d9
I20260928 20:19:53.025226 10546 ts_tablet_manager.cc:1939] T fd5e92ec6f894b70a9b2c8fe743159d9 P 39e67e2773bb4225a13ec3d8425a088a: Deleting consensus metadata
I20260928 20:19:53.026093 10547 ts_tablet_manager.cc:1918] T fd5e92ec6f894b70a9b2c8fe743159d9 P 14a5194f37a543ad9cf9c8130a5d11af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:19:53.026152 10547 log.cc:1196] T fd5e92ec6f894b70a9b2c8fe743159d9 P 14a5194f37a543ad9cf9c8130a5d11af: Deleting WAL directory at /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-1/wal/wals/fd5e92ec6f894b70a9b2c8fe743159d9
I20260928 20:19:53.026338 10547 ts_tablet_manager.cc:1939] T fd5e92ec6f894b70a9b2c8fe743159d9 P 14a5194f37a543ad9cf9c8130a5d11af: Deleting consensus metadata
W20260928 20:19:53.030552 10246 connection.cc:535] server connection from 127.0.0.1:43402 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 20:19:53.030751 10379 connection.cc:535] server connection from 127.0.0.1:53470 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 20:19:53.031426 10106 connection.cc:535] server connection from 127.0.0.1:59926 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 20:19:53.035939 10507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:53.021918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60362 (local address 127.3.18.60:38323)
0928 20:19:53.021961 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:53.021963 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:53.028015 (+ 6052us) server_negotiation.cc:407] Connection header received
0928 20:19:53.028066 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:53.028067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:53.028078 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:53.028134 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:53.028759 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:53.029302 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:53.030115 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:53.030420 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:53.030487 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:53.032336 (+ 1849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:53.032338 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:53.032338 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:53.032347 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:53.034281 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:53.034937 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:53.034938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:53.034939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:53.035005 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:53.035252 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:53.035253 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:53.035254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:53.035395 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:53.035511 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:53.035707 (+ 196us) server_negotiation.cc:299] Negotiation successful
0928 20:19:53.035821 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 20:19:53.041752 10506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:53.033077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57486 (local address 127.3.18.61:33799)
0928 20:19:53.033119 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:53.033121 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:53.033474 (+ 353us) server_negotiation.cc:407] Connection header received
0928 20:19:53.033586 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:53.033587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:53.033600 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:53.033665 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:53.034265 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:53.035053 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:53.037047 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:53.037270 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:53.037356 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:53.038733 (+ 1377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:53.038735 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:53.038735 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:53.038742 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:53.040503 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:53.040866 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:53.040868 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:53.040868 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:53.040936 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:53.041133 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:53.041134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:53.041134 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:53.041255 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:53.041367 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:53.041547 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 20:19:53.041629 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 20:19:53.042152 10225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:53.034597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60372 (local address 127.3.18.60:38323)
0928 20:19:53.034802 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:53.034804 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:53.034811 (+ 7us) server_negotiation.cc:407] Connection header received
0928 20:19:53.034831 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:53.034831 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:53.034842 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:53.034890 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:53.035524 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:53.036281 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:53.037378 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:53.037697 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:53.037739 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:53.039174 (+ 1435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:53.039175 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:53.039175 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:53.039182 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:53.040936 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:53.041307 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:53.041309 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:53.041309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:53.041363 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:53.041596 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:53.041597 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:53.041597 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:53.041722 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:53.041790 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:53.041986 (+ 196us) server_negotiation.cc:299] Negotiation successful
0928 20:19:53.042059 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184}
I20260928 20:19:54.018249 10556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:54.017992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:42379 (local address 127.3.18.2:58255)
0928 20:19:54.018120 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:54.018203 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:42379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":80,"threads_started":1}
W20260928 20:19:54.018476 10357 heartbeater.cc:644] Failed to heartbeat to 127.3.18.62:42379 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.18.62:42379: connect: Connection refused (error 111)
I20260928 20:19:54.019629 10557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:54.019362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:42379 (local address 127.3.18.3:45967)
0928 20:19:54.019499 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:54.019579 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:42379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":139,"threads_started":1}
I20260928 20:19:54.019742 10558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:54.019487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:42379 (local address 127.3.18.1:53881)
0928 20:19:54.019599 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:54.019689 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:42379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":70,"threads_started":1}
W20260928 20:19:54.019899 10493 heartbeater.cc:644] Failed to heartbeat to 127.3.18.62:42379 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.18.62:42379: connect: Connection refused (error 111)
W20260928 20:19:54.019961 10220 heartbeater.cc:644] Failed to heartbeat to 127.3.18.62:42379 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.18.62:42379: connect: Connection refused (error 111)
I20260928 20:19:54.520792 10559 raft_consensus.cc:482] T 00000000000000000000000000000000 P 645a6ba8e676428585dc367ef37372c4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ef4554cccf754a53a7d74f685421731a)
I20260928 20:19:54.521003 10559 raft_consensus.cc:504] T 00000000000000000000000000000000 P 645a6ba8e676428585dc367ef37372c4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ef4554cccf754a53a7d74f685421731a" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 42379 } } peers { permanent_uuid: "d13b62b954784d05bd09150e14f708b7" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 33799 } attrs { promote: false } } peers { permanent_uuid: "645a6ba8e676428585dc367ef37372c4" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 38323 } attrs { promote: false } }
I20260928 20:19:54.521353 10559 leader_election.cc:288] T 00000000000000000000000000000000 P 645a6ba8e676428585dc367ef37372c4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ef4554cccf754a53a7d74f685421731a (127.3.18.62:42379), d13b62b954784d05bd09150e14f708b7 (127.3.18.61:33799)
I20260928 20:19:54.521726 10560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:54.521522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:42379 (local address 127.0.0.1:35714)
0928 20:19:54.521614 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:54.521675 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:42379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72,"thread_start_us":51,"threads_started":1}
I20260928 20:19:54.521924 10563 raft_consensus.cc:482] T 00000000000000000000000000000000 P d13b62b954784d05bd09150e14f708b7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ef4554cccf754a53a7d74f685421731a)
W20260928 20:19:54.521948 10026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.18.62:42379: connect: Connection refused (error 111)
I20260928 20:19:54.522058 10563 raft_consensus.cc:504] T 00000000000000000000000000000000 P d13b62b954784d05bd09150e14f708b7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ef4554cccf754a53a7d74f685421731a" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 42379 } } peers { permanent_uuid: "d13b62b954784d05bd09150e14f708b7" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 33799 } attrs { promote: false } } peers { permanent_uuid: "645a6ba8e676428585dc367ef37372c4" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 38323 } attrs { promote: false } }
I20260928 20:19:54.522374 10560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:54.522242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:42379 (local address 127.0.0.1:35716)
0928 20:19:54.522287 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:54.522324 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:42379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 20:19:54.522367 10563 leader_election.cc:288] T 00000000000000000000000000000000 P d13b62b954784d05bd09150e14f708b7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ef4554cccf754a53a7d74f685421731a (127.3.18.62:42379), 645a6ba8e676428585dc367ef37372c4 (127.3.18.60:38323)
W20260928 20:19:54.522459 10026 leader_election.cc:334] T 00000000000000000000000000000000 P 645a6ba8e676428585dc367ef37372c4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ef4554cccf754a53a7d74f685421731a (127.3.18.62:42379): Network error: Client connection negotiation failed: client connection to 127.3.18.62:42379: connect: Connection refused (error 111)
I20260928 20:19:54.522830 10566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:54.522545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:42379 (local address 127.0.0.1:35732)
0928 20:19:54.522734 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:54.522790 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:42379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":60,"threads_started":1}
W20260928 20:19:54.522925 9840 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.18.62:42379: connect: Connection refused (error 111)
I20260928 20:19:54.523384 10566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:54.523263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:42379 (local address 127.0.0.1:35738)
0928 20:19:54.523322 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:54.523349 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:42379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 20:19:54.523490 9840 leader_election.cc:334] T 00000000000000000000000000000000 P d13b62b954784d05bd09150e14f708b7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ef4554cccf754a53a7d74f685421731a (127.3.18.62:42379): Network error: Client connection negotiation failed: client connection to 127.3.18.62:42379: connect: Connection refused (error 111)
I20260928 20:19:54.525420 10561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:54.521552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.61:33799 (local address 127.0.0.1:57494)
0928 20:19:54.521632 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:54.521643 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:54.521706 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:54.521973 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:54.521974 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:54.521977 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:19:54.522081 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:54.522085 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:54.523126 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:54.523127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:54.524439 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:54.524441 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:54.525210 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:54.525211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:54.525268 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:54.525272 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:19:54.525331 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 20:19:54.525367 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69,"thread_start_us":52,"threads_started":1}
I20260928 20:19:54.525454 10562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:54.521600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57494 (local address 127.3.18.61:33799)
0928 20:19:54.521809 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:54.521812 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:54.521824 (+ 12us) server_negotiation.cc:407] Connection header received
0928 20:19:54.521858 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:54.521859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:54.521872 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:54.521956 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:19:54.522164 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:54.523029 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:54.524538 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:54.525095 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:54.525176 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:54.525189 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:54.525351 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 20:19:54.525393 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":57,"threads_started":1}
I20260928 20:19:54.525734 9864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "645a6ba8e676428585dc367ef37372c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d13b62b954784d05bd09150e14f708b7" is_pre_election: true
I20260928 20:19:54.525826 9864 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d13b62b954784d05bd09150e14f708b7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 645a6ba8e676428585dc367ef37372c4 in term 1.
I20260928 20:19:54.526015 10024 leader_election.cc:302] T 00000000000000000000000000000000 P 645a6ba8e676428585dc367ef37372c4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 645a6ba8e676428585dc367ef37372c4, d13b62b954784d05bd09150e14f708b7; no voters: ef4554cccf754a53a7d74f685421731a
I20260928 20:19:54.526131 10559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 645a6ba8e676428585dc367ef37372c4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 20:19:54.526171 10559 raft_consensus.cc:482] T 00000000000000000000000000000000 P 645a6ba8e676428585dc367ef37372c4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ef4554cccf754a53a7d74f685421731a)
I20260928 20:19:54.526201 10559 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 645a6ba8e676428585dc367ef37372c4 [term 1 FOLLOWER]: Advancing to term 2
I20260928 20:19:54.526623 10565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:54.522587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.60:38323 (local address 127.0.0.1:60378)
0928 20:19:54.522950 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:54.522959 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:54.523019 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:54.523217 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:54.523219 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:54.523220 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:19:54.523261 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:54.523262 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:54.524264 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:54.524265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:54.525488 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:54.525490 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:54.526375 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:54.526377 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:54.526454 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:54.526459 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 20:19:54.526522 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 20:19:54.526562 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":54,"threads_started":1}
I20260928 20:19:54.526716 10567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:54.522789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60378 (local address 127.3.18.60:38323)
0928 20:19:54.522911 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:54.522913 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:54.523078 (+ 165us) server_negotiation.cc:407] Connection header received
0928 20:19:54.523104 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:54.523105 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:54.523116 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:54.523178 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:19:54.523332 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:54.524172 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:54.525694 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:54.526294 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:54.526354 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:54.526363 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:54.526566 (+ 203us) server_negotiation.cc:299] Negotiation successful
0928 20:19:54.526612 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":75,"threads_started":1}
I20260928 20:19:54.526924 10050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d13b62b954784d05bd09150e14f708b7" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "645a6ba8e676428585dc367ef37372c4" is_pre_election: true
I20260928 20:19:54.527384 10559 raft_consensus.cc:504] T 00000000000000000000000000000000 P 645a6ba8e676428585dc367ef37372c4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ef4554cccf754a53a7d74f685421731a" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 42379 } } peers { permanent_uuid: "d13b62b954784d05bd09150e14f708b7" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 33799 } attrs { promote: false } } peers { permanent_uuid: "645a6ba8e676428585dc367ef37372c4" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 38323 } attrs { promote: false } }
I20260928 20:19:54.527576 10050 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 645a6ba8e676428585dc367ef37372c4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d13b62b954784d05bd09150e14f708b7 in current term 2: Already voted for candidate 645a6ba8e676428585dc367ef37372c4 in this term.
I20260928 20:19:54.527706 10559 leader_election.cc:288] T 00000000000000000000000000000000 P 645a6ba8e676428585dc367ef37372c4 [CANDIDATE]: Term 2 election: Requested vote from peers ef4554cccf754a53a7d74f685421731a (127.3.18.62:42379), d13b62b954784d05bd09150e14f708b7 (127.3.18.61:33799)
I20260928 20:19:54.527868 10561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:54.527622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:42379 (local address 127.0.0.1:35752)
0928 20:19:54.527817 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:54.527845 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:42379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180}
I20260928 20:19:54.527863 9841 leader_election.cc:302] T 00000000000000000000000000000000 P d13b62b954784d05bd09150e14f708b7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d13b62b954784d05bd09150e14f708b7; no voters: 645a6ba8e676428585dc367ef37372c4, ef4554cccf754a53a7d74f685421731a
I20260928 20:19:54.527879 9864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "645a6ba8e676428585dc367ef37372c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d13b62b954784d05bd09150e14f708b7"
I20260928 20:19:54.527980 10563 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d13b62b954784d05bd09150e14f708b7 [term 1 FOLLOWER]: Advancing to term 2
I20260928 20:19:54.528321 10561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:54.528200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:42379 (local address 127.0.0.1:35754)
0928 20:19:54.528259 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:54.528289 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:42379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 20:19:54.528409 10026 leader_election.cc:334] T 00000000000000000000000000000000 P 645a6ba8e676428585dc367ef37372c4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ef4554cccf754a53a7d74f685421731a (127.3.18.62:42379): Network error: Client connection negotiation failed: client connection to 127.3.18.62:42379: connect: Connection refused (error 111)
I20260928 20:19:54.528808 10563 raft_consensus.cc:2726] T 00000000000000000000000000000000 P d13b62b954784d05bd09150e14f708b7 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 20:19:54.529524 9864 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d13b62b954784d05bd09150e14f708b7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 645a6ba8e676428585dc367ef37372c4 in term 2.
I20260928 20:19:54.530028 10024 leader_election.cc:302] T 00000000000000000000000000000000 P 645a6ba8e676428585dc367ef37372c4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 645a6ba8e676428585dc367ef37372c4, d13b62b954784d05bd09150e14f708b7; no voters: ef4554cccf754a53a7d74f685421731a
I20260928 20:19:54.530112 10559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 645a6ba8e676428585dc367ef37372c4 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 20:19:54.530278 10559 raft_consensus.cc:686] T 00000000000000000000000000000000 P 645a6ba8e676428585dc367ef37372c4 [term 2 LEADER]: Becoming Leader. State: Replica: 645a6ba8e676428585dc367ef37372c4, State: Running, Role: LEADER
I20260928 20:19:54.530359 10559 consensus_queue.cc:237] T 00000000000000000000000000000000 P 645a6ba8e676428585dc367ef37372c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ef4554cccf754a53a7d74f685421731a" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 42379 } } peers { permanent_uuid: "d13b62b954784d05bd09150e14f708b7" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 33799 } attrs { promote: false } } peers { permanent_uuid: "645a6ba8e676428585dc367ef37372c4" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 38323 } attrs { promote: false } }
I20260928 20:19:54.530704 10569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 645a6ba8e676428585dc367ef37372c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 645a6ba8e676428585dc367ef37372c4. Latest consensus state: current_term: 2 leader_uuid: "645a6ba8e676428585dc367ef37372c4" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ef4554cccf754a53a7d74f685421731a" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 42379 } } peers { permanent_uuid: "d13b62b954784d05bd09150e14f708b7" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 33799 } attrs { promote: false } } peers { permanent_uuid: "645a6ba8e676428585dc367ef37372c4" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 38323 } attrs { promote: false } } }
I20260928 20:19:54.530810 10569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 645a6ba8e676428585dc367ef37372c4 [sys.catalog]: This master's current role is: LEADER
I20260928 20:19:54.531010 10571 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:19:54.531381 10571 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:19:54.531481 10571 catalog_manager.cc:1259] Loaded cluster ID: 44e10b6ab55043b4b4c53ad9882d8ab4
I20260928 20:19:54.531536 10571 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:19:54.531714 10571 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:19:54.531841 10571 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 645a6ba8e676428585dc367ef37372c4: Loaded TSK: 0
I20260928 20:19:54.531991 10571 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 20:19:54.586653 10040 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60372:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 20:19:54.588516 10040 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 20:19:54.589643 10561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:54.589490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:42379 (local address 127.0.0.1:35762)
0928 20:19:54.589569 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:54.589612 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:42379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 20:19:54.589722 9864 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d13b62b954784d05bd09150e14f708b7 [term 2 FOLLOWER]: Refusing update from remote peer 645a6ba8e676428585dc367ef37372c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260928 20:19:54.590070 10561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:54.589906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:42379 (local address 127.0.0.1:35776)
0928 20:19:54.589948 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:54.590014 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:42379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260928 20:19:54.590024 10569 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 645a6ba8e676428585dc367ef37372c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13b62b954784d05bd09150e14f708b7" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 33799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260928 20:19:54.590157 10026 consensus_peers.cc:487] T 00000000000000000000000000000000 P 645a6ba8e676428585dc367ef37372c4 -> Peer ef4554cccf754a53a7d74f685421731a (127.3.18.62:42379): Couldn't send request to peer ef4554cccf754a53a7d74f685421731a. Status: Network error: Client connection negotiation failed: client connection to 127.3.18.62:42379: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 20:19:54.590775 10563 sys_catalog.cc:455] T 00000000000000000000000000000000 P d13b62b954784d05bd09150e14f708b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 645a6ba8e676428585dc367ef37372c4. Latest consensus state: current_term: 2 leader_uuid: "645a6ba8e676428585dc367ef37372c4" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ef4554cccf754a53a7d74f685421731a" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 42379 } } peers { permanent_uuid: "d13b62b954784d05bd09150e14f708b7" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 33799 } attrs { promote: false } } peers { permanent_uuid: "645a6ba8e676428585dc367ef37372c4" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 38323 } attrs { promote: false } } }
I20260928 20:19:54.590893 10563 sys_catalog.cc:458] T 00000000000000000000000000000000 P d13b62b954784d05bd09150e14f708b7 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 20:19:54.591336 10569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 645a6ba8e676428585dc367ef37372c4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "645a6ba8e676428585dc367ef37372c4" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ef4554cccf754a53a7d74f685421731a" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 42379 } } peers { permanent_uuid: "d13b62b954784d05bd09150e14f708b7" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 33799 } attrs { promote: false } } peers { permanent_uuid: "645a6ba8e676428585dc367ef37372c4" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 38323 } attrs { promote: false } } }
I20260928 20:19:54.591436 10569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 645a6ba8e676428585dc367ef37372c4 [sys.catalog]: This master's current role is: LEADER
I20260928 20:19:54.591820 10563 sys_catalog.cc:455] T 00000000000000000000000000000000 P d13b62b954784d05bd09150e14f708b7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "645a6ba8e676428585dc367ef37372c4" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ef4554cccf754a53a7d74f685421731a" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 42379 } } peers { permanent_uuid: "d13b62b954784d05bd09150e14f708b7" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 33799 } attrs { promote: false } } peers { permanent_uuid: "645a6ba8e676428585dc367ef37372c4" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 38323 } attrs { promote: false } } }
I20260928 20:19:54.591919 10563 sys_catalog.cc:458] T 00000000000000000000000000000000 P d13b62b954784d05bd09150e14f708b7 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 20:19:54.596966 10560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:54.593529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.1:44945 (local address 127.0.0.1:59932)
0928 20:19:54.593674 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:54.593686 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:54.593745 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:54.594048 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:54.594049 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:54.594052 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:19:54.594116 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:54.594117 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:54.595090 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:54.595091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:54.596099 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:54.596102 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:54.596703 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:54.596704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:54.596774 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:54.596779 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 20:19:54.596840 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 20:19:54.596884 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":80,"thread_start_us":90,"threads_started":1}
I20260928 20:19:54.596980 10579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:54.593736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59932 (local address 127.3.18.1:44945)
0928 20:19:54.593839 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:54.593841 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:54.593848 (+ 7us) server_negotiation.cc:407] Connection header received
0928 20:19:54.593872 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:54.593873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:54.593889 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:54.594186 (+ 297us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:19:54.594231 (+ 45us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:54.594994 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:54.596187 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:54.596617 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:54.596691 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:54.596704 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:54.596854 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 20:19:54.596898 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":61,"threads_started":1}
I20260928 20:19:54.597293 10153 tablet_service.cc:1467] Processing CreateTablet for tablet c965d9b923c141eb8600b166fb2df2f1 (DEFAULT_TABLE table=test-table [id=8c9b2e8ed370455db41cd6a1b4988ee5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:19:54.597429 10153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c965d9b923c141eb8600b166fb2df2f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:54.597611 10577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:54.593518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.2:35465 (local address 127.0.0.1:43416)
0928 20:19:54.593663 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:54.593673 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:54.593724 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:54.594199 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:54.594200 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:54.594202 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:19:54.594258 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:54.594260 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:54.595205 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:54.595205 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:54.596478 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:54.596480 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:54.597336 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:54.597337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:54.597405 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:54.597410 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 20:19:54.597477 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 20:19:54.597531 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":1}
I20260928 20:19:54.597913 10561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:54.593403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.3:38189 (local address 127.0.0.1:53482)
0928 20:19:54.593458 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:54.593470 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:54.593511 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:54.594112 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:54.594112 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:54.594113 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:19:54.594142 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:54.594143 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:54.595087 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:54.595088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:54.596449 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:54.596462 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:54.597725 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:54.597727 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:54.597781 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:54.597786 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 20:19:54.597815 (+ 29us) client_negotiation.cc:240] Negotiation successful
0928 20:19:54.597852 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":57}
I20260928 20:19:54.597913 10578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:54.593678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43416 (local address 127.3.18.2:35465)
0928 20:19:54.594035 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:54.594037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:54.594046 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:19:54.594077 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:54.594078 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:54.594090 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:54.594191 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:19:54.594363 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:54.594993 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:54.596580 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:54.597232 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:54.597333 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:54.597346 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:54.597790 (+ 444us) server_negotiation.cc:299] Negotiation successful
0928 20:19:54.597840 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":82,"threads_started":1}
I20260928 20:19:54.598265 10580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:54.593840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53482 (local address 127.3.18.3:38189)
0928 20:19:54.593977 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:54.593980 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:54.593987 (+ 7us) server_negotiation.cc:407] Connection header received
0928 20:19:54.594013 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:54.594014 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:54.594025 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:54.594203 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:19:54.594243 (+ 40us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:54.594989 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:54.596564 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:54.597627 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:54.598085 (+ 458us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:54.598104 (+ 19us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:54.598139 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 20:19:54.598193 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":67,"threads_started":1}
I20260928 20:19:54.598275 10290 tablet_service.cc:1467] Processing CreateTablet for tablet c965d9b923c141eb8600b166fb2df2f1 (DEFAULT_TABLE table=test-table [id=8c9b2e8ed370455db41cd6a1b4988ee5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:19:54.598412 10290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c965d9b923c141eb8600b166fb2df2f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:54.598520 10426 tablet_service.cc:1467] Processing CreateTablet for tablet c965d9b923c141eb8600b166fb2df2f1 (DEFAULT_TABLE table=test-table [id=8c9b2e8ed370455db41cd6a1b4988ee5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:19:54.598623 10426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c965d9b923c141eb8600b166fb2df2f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:54.599144 10581 tablet_bootstrap.cc:492] T c965d9b923c141eb8600b166fb2df2f1 P 39e67e2773bb4225a13ec3d8425a088a: Bootstrap starting.
I20260928 20:19:54.599514 10581 tablet_bootstrap.cc:654] T c965d9b923c141eb8600b166fb2df2f1 P 39e67e2773bb4225a13ec3d8425a088a: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:54.599852 10582 tablet_bootstrap.cc:492] T c965d9b923c141eb8600b166fb2df2f1 P 14a5194f37a543ad9cf9c8130a5d11af: Bootstrap starting.
I20260928 20:19:54.599923 10584 tablet_bootstrap.cc:492] T c965d9b923c141eb8600b166fb2df2f1 P 1f0e35e0a8fd4cec9d631c0701e0b76f: Bootstrap starting.
I20260928 20:19:54.600242 10582 tablet_bootstrap.cc:654] T c965d9b923c141eb8600b166fb2df2f1 P 14a5194f37a543ad9cf9c8130a5d11af: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:54.600322 10584 tablet_bootstrap.cc:654] T c965d9b923c141eb8600b166fb2df2f1 P 1f0e35e0a8fd4cec9d631c0701e0b76f: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:54.600762 10582 tablet_bootstrap.cc:492] T c965d9b923c141eb8600b166fb2df2f1 P 14a5194f37a543ad9cf9c8130a5d11af: No bootstrap required, opened a new log
I20260928 20:19:54.600832 10582 ts_tablet_manager.cc:1397] T c965d9b923c141eb8600b166fb2df2f1 P 14a5194f37a543ad9cf9c8130a5d11af: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 20:19:54.600952 10584 tablet_bootstrap.cc:492] T c965d9b923c141eb8600b166fb2df2f1 P 1f0e35e0a8fd4cec9d631c0701e0b76f: No bootstrap required, opened a new log
I20260928 20:19:54.600971 10582 raft_consensus.cc:348] T c965d9b923c141eb8600b166fb2df2f1 P 14a5194f37a543ad9cf9c8130a5d11af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0e35e0a8fd4cec9d631c0701e0b76f" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38189 } } peers { permanent_uuid: "39e67e2773bb4225a13ec3d8425a088a" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 44945 } } peers { permanent_uuid: "14a5194f37a543ad9cf9c8130a5d11af" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 35465 } }
I20260928 20:19:54.601049 10584 ts_tablet_manager.cc:1397] T c965d9b923c141eb8600b166fb2df2f1 P 1f0e35e0a8fd4cec9d631c0701e0b76f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 20:19:54.601060 10582 raft_consensus.cc:374] T c965d9b923c141eb8600b166fb2df2f1 P 14a5194f37a543ad9cf9c8130a5d11af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:54.601085 10582 raft_consensus.cc:729] T c965d9b923c141eb8600b166fb2df2f1 P 14a5194f37a543ad9cf9c8130a5d11af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14a5194f37a543ad9cf9c8130a5d11af, State: Initialized, Role: FOLLOWER
I20260928 20:19:54.601138 10582 consensus_queue.cc:260] T c965d9b923c141eb8600b166fb2df2f1 P 14a5194f37a543ad9cf9c8130a5d11af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0e35e0a8fd4cec9d631c0701e0b76f" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38189 } } peers { permanent_uuid: "39e67e2773bb4225a13ec3d8425a088a" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 44945 } } peers { permanent_uuid: "14a5194f37a543ad9cf9c8130a5d11af" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 35465 } }
I20260928 20:19:54.601284 10582 ts_tablet_manager.cc:1428] T c965d9b923c141eb8600b166fb2df2f1 P 14a5194f37a543ad9cf9c8130a5d11af: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 20:19:54.601513 10581 tablet_bootstrap.cc:492] T c965d9b923c141eb8600b166fb2df2f1 P 39e67e2773bb4225a13ec3d8425a088a: No bootstrap required, opened a new log
I20260928 20:19:54.601572 10581 ts_tablet_manager.cc:1397] T c965d9b923c141eb8600b166fb2df2f1 P 39e67e2773bb4225a13ec3d8425a088a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 20:19:54.601703 10581 raft_consensus.cc:348] T c965d9b923c141eb8600b166fb2df2f1 P 39e67e2773bb4225a13ec3d8425a088a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0e35e0a8fd4cec9d631c0701e0b76f" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38189 } } peers { permanent_uuid: "39e67e2773bb4225a13ec3d8425a088a" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 44945 } } peers { permanent_uuid: "14a5194f37a543ad9cf9c8130a5d11af" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 35465 } }
I20260928 20:19:54.601790 10581 raft_consensus.cc:374] T c965d9b923c141eb8600b166fb2df2f1 P 39e67e2773bb4225a13ec3d8425a088a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:54.601822 10581 raft_consensus.cc:729] T c965d9b923c141eb8600b166fb2df2f1 P 39e67e2773bb4225a13ec3d8425a088a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39e67e2773bb4225a13ec3d8425a088a, State: Initialized, Role: FOLLOWER
I20260928 20:19:54.601869 10581 consensus_queue.cc:260] T c965d9b923c141eb8600b166fb2df2f1 P 39e67e2773bb4225a13ec3d8425a088a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0e35e0a8fd4cec9d631c0701e0b76f" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38189 } } peers { permanent_uuid: "39e67e2773bb4225a13ec3d8425a088a" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 44945 } } peers { permanent_uuid: "14a5194f37a543ad9cf9c8130a5d11af" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 35465 } }
I20260928 20:19:54.602062 10581 ts_tablet_manager.cc:1428] T c965d9b923c141eb8600b166fb2df2f1 P 39e67e2773bb4225a13ec3d8425a088a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 20:19:54.602082 10588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:54.601591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:42379 (local address 127.3.18.2:38073)
0928 20:19:54.601738 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:54.601779 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:42379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":88,"threads_started":1}
I20260928 20:19:54.602167 10356 heartbeater.cc:503] Master 127.3.18.60:38323 was elected leader, sending a full tablet report...
I20260928 20:19:54.602674 10590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:54.602461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:42379 (local address 127.3.18.1:52957)
0928 20:19:54.602592 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:54.602628 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:42379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":79,"threads_started":1}
I20260928 20:19:54.602890 10584 raft_consensus.cc:348] T c965d9b923c141eb8600b166fb2df2f1 P 1f0e35e0a8fd4cec9d631c0701e0b76f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0e35e0a8fd4cec9d631c0701e0b76f" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38189 } } peers { permanent_uuid: "39e67e2773bb4225a13ec3d8425a088a" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 44945 } } peers { permanent_uuid: "14a5194f37a543ad9cf9c8130a5d11af" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 35465 } }
I20260928 20:19:54.602999 10584 raft_consensus.cc:374] T c965d9b923c141eb8600b166fb2df2f1 P 1f0e35e0a8fd4cec9d631c0701e0b76f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:54.603036 10584 raft_consensus.cc:729] T c965d9b923c141eb8600b166fb2df2f1 P 1f0e35e0a8fd4cec9d631c0701e0b76f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0e35e0a8fd4cec9d631c0701e0b76f, State: Initialized, Role: FOLLOWER
I20260928 20:19:54.603097 10584 consensus_queue.cc:260] T c965d9b923c141eb8600b166fb2df2f1 P 1f0e35e0a8fd4cec9d631c0701e0b76f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0e35e0a8fd4cec9d631c0701e0b76f" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38189 } } peers { permanent_uuid: "39e67e2773bb4225a13ec3d8425a088a" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 44945 } } peers { permanent_uuid: "14a5194f37a543ad9cf9c8130a5d11af" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 35465 } }
I20260928 20:19:54.603294 10584 ts_tablet_manager.cc:1428] T c965d9b923c141eb8600b166fb2df2f1 P 1f0e35e0a8fd4cec9d631c0701e0b76f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260928 20:19:54.603440 10219 heartbeater.cc:503] Master 127.3.18.60:38323 was elected leader, sending a full tablet report...
I20260928 20:19:54.603909 10492 heartbeater.cc:503] Master 127.3.18.60:38323 was elected leader, sending a full tablet report...
I20260928 20:19:54.604010 10590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:54.603887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:42379 (local address 127.3.18.1:33337)
0928 20:19:54.603955 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:54.603984 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:42379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 20:19:54.604250 10592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:54.603437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:42379 (local address 127.3.18.3:40105)
0928 20:19:54.604168 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:54.604207 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:42379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":706,"thread_start_us":82,"threads_started":1}
I20260928 20:19:54.604550 10592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:54.604473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:42379 (local address 127.3.18.3:52663)
0928 20:19:54.604512 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:54.604536 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:42379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260928 20:19:54.659793 10587 raft_consensus.cc:482] T c965d9b923c141eb8600b166fb2df2f1 P 14a5194f37a543ad9cf9c8130a5d11af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:19:54.659894 10587 raft_consensus.cc:504] T c965d9b923c141eb8600b166fb2df2f1 P 14a5194f37a543ad9cf9c8130a5d11af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0e35e0a8fd4cec9d631c0701e0b76f" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38189 } } peers { permanent_uuid: "39e67e2773bb4225a13ec3d8425a088a" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 44945 } } peers { permanent_uuid: "14a5194f37a543ad9cf9c8130a5d11af" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 35465 } }
I20260928 20:19:54.660205 10587 leader_election.cc:288] T c965d9b923c141eb8600b166fb2df2f1 P 14a5194f37a543ad9cf9c8130a5d11af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f0e35e0a8fd4cec9d631c0701e0b76f (127.3.18.3:38189), 39e67e2773bb4225a13ec3d8425a088a (127.3.18.1:44945)
I20260928 20:19:54.664644 10580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:54.660534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:42707 (local address 127.3.18.3:38189)
0928 20:19:54.660586 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:54.660588 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:54.660599 (+ 11us) server_negotiation.cc:407] Connection header received
0928 20:19:54.660630 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:54.660631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:54.660648 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:54.660731 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:19:54.660933 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:54.661756 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:54.663598 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:54.664212 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:54.664294 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:54.664309 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:54.664505 (+ 196us) server_negotiation.cc:299] Negotiation successful
0928 20:19:54.664548 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 20:19:54.664644 10588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:54.660403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.3:38189 (local address 127.3.18.2:42707)
0928 20:19:54.660483 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:54.660496 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:54.660572 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:54.660759 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:54.660761 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:54.660763 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:19:54.660843 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:54.660845 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:54.661873 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:54.661874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:54.663482 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:54.663486 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:54.664304 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:54.664305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:54.664412 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:54.664417 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 20:19:54.664487 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 20:19:54.664536 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":115}
I20260928 20:19:54.664739 10579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:54.660503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:59029 (local address 127.3.18.1:44945)
0928 20:19:54.660552 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:54.660554 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:54.660715 (+ 161us) server_negotiation.cc:407] Connection header received
0928 20:19:54.660778 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:54.660779 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:54.660798 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:54.660840 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:19:54.660989 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:54.661728 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:54.663661 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:54.664289 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:54.664364 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:54.664375 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:54.664622 (+ 247us) server_negotiation.cc:299] Negotiation successful
0928 20:19:54.664671 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 20:19:54.664768 10593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:54.660403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.1:44945 (local address 127.3.18.2:59029)
0928 20:19:54.660623 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:54.660635 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:54.660698 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:54.660864 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:54.660865 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:54.660866 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:19:54.660909 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:54.660910 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:54.661873 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:54.661874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:54.663569 (+ 1695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:54.663572 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:54.664401 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:54.664402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:54.664535 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:54.664539 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:19:54.664599 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 20:19:54.664634 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":136,"thread_start_us":112,"threads_started":1}
I20260928 20:19:54.665022 10173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c965d9b923c141eb8600b166fb2df2f1" candidate_uuid: "14a5194f37a543ad9cf9c8130a5d11af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39e67e2773bb4225a13ec3d8425a088a" is_pre_election: true
I20260928 20:19:54.665002 10446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c965d9b923c141eb8600b166fb2df2f1" candidate_uuid: "14a5194f37a543ad9cf9c8130a5d11af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0e35e0a8fd4cec9d631c0701e0b76f" is_pre_election: true
I20260928 20:19:54.665126 10173 raft_consensus.cc:2454] T c965d9b923c141eb8600b166fb2df2f1 P 39e67e2773bb4225a13ec3d8425a088a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14a5194f37a543ad9cf9c8130a5d11af in term 0.
I20260928 20:19:54.665144 10446 raft_consensus.cc:2454] T c965d9b923c141eb8600b166fb2df2f1 P 1f0e35e0a8fd4cec9d631c0701e0b76f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14a5194f37a543ad9cf9c8130a5d11af in term 0.
I20260928 20:19:54.665359 10245 leader_election.cc:302] T c965d9b923c141eb8600b166fb2df2f1 P 14a5194f37a543ad9cf9c8130a5d11af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14a5194f37a543ad9cf9c8130a5d11af, 39e67e2773bb4225a13ec3d8425a088a; no voters:
I20260928 20:19:54.665544 10587 raft_consensus.cc:2781] T c965d9b923c141eb8600b166fb2df2f1 P 14a5194f37a543ad9cf9c8130a5d11af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 20:19:54.665589 10587 raft_consensus.cc:482] T c965d9b923c141eb8600b166fb2df2f1 P 14a5194f37a543ad9cf9c8130a5d11af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 20:19:54.665616 10587 raft_consensus.cc:3037] T c965d9b923c141eb8600b166fb2df2f1 P 14a5194f37a543ad9cf9c8130a5d11af [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:54.666432 10587 raft_consensus.cc:504] T c965d9b923c141eb8600b166fb2df2f1 P 14a5194f37a543ad9cf9c8130a5d11af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0e35e0a8fd4cec9d631c0701e0b76f" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38189 } } peers { permanent_uuid: "39e67e2773bb4225a13ec3d8425a088a" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 44945 } } peers { permanent_uuid: "14a5194f37a543ad9cf9c8130a5d11af" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 35465 } }
I20260928 20:19:54.666607 10587 leader_election.cc:288] T c965d9b923c141eb8600b166fb2df2f1 P 14a5194f37a543ad9cf9c8130a5d11af [CANDIDATE]: Term 1 election: Requested vote from peers 1f0e35e0a8fd4cec9d631c0701e0b76f (127.3.18.3:38189), 39e67e2773bb4225a13ec3d8425a088a (127.3.18.1:44945)
I20260928 20:19:54.666760 10446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c965d9b923c141eb8600b166fb2df2f1" candidate_uuid: "14a5194f37a543ad9cf9c8130a5d11af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0e35e0a8fd4cec9d631c0701e0b76f"
I20260928 20:19:54.666771 10173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c965d9b923c141eb8600b166fb2df2f1" candidate_uuid: "14a5194f37a543ad9cf9c8130a5d11af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39e67e2773bb4225a13ec3d8425a088a"
I20260928 20:19:54.666846 10173 raft_consensus.cc:3037] T c965d9b923c141eb8600b166fb2df2f1 P 39e67e2773bb4225a13ec3d8425a088a [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:54.666846 10446 raft_consensus.cc:3037] T c965d9b923c141eb8600b166fb2df2f1 P 1f0e35e0a8fd4cec9d631c0701e0b76f [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:54.667692 10173 raft_consensus.cc:2454] T c965d9b923c141eb8600b166fb2df2f1 P 39e67e2773bb4225a13ec3d8425a088a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14a5194f37a543ad9cf9c8130a5d11af in term 1.
I20260928 20:19:54.667737 10446 raft_consensus.cc:2454] T c965d9b923c141eb8600b166fb2df2f1 P 1f0e35e0a8fd4cec9d631c0701e0b76f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14a5194f37a543ad9cf9c8130a5d11af in term 1.
I20260928 20:19:54.667907 10245 leader_election.cc:302] T c965d9b923c141eb8600b166fb2df2f1 P 14a5194f37a543ad9cf9c8130a5d11af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14a5194f37a543ad9cf9c8130a5d11af, 39e67e2773bb4225a13ec3d8425a088a; no voters:
I20260928 20:19:54.668018 10587 raft_consensus.cc:2781] T c965d9b923c141eb8600b166fb2df2f1 P 14a5194f37a543ad9cf9c8130a5d11af [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:19:54.668195 10587 raft_consensus.cc:686] T c965d9b923c141eb8600b166fb2df2f1 P 14a5194f37a543ad9cf9c8130a5d11af [term 1 LEADER]: Becoming Leader. State: Replica: 14a5194f37a543ad9cf9c8130a5d11af, State: Running, Role: LEADER
I20260928 20:19:54.668293 10587 consensus_queue.cc:237] T c965d9b923c141eb8600b166fb2df2f1 P 14a5194f37a543ad9cf9c8130a5d11af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0e35e0a8fd4cec9d631c0701e0b76f" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38189 } } peers { permanent_uuid: "39e67e2773bb4225a13ec3d8425a088a" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 44945 } } peers { permanent_uuid: "14a5194f37a543ad9cf9c8130a5d11af" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 35465 } }
I20260928 20:19:54.668747 10588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:54.668575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:42379 (local address 127.3.18.2:59439)
0928 20:19:54.668660 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:54.668706 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:42379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260928 20:19:54.669356 10040 catalog_manager.cc:5495] T c965d9b923c141eb8600b166fb2df2f1 P 14a5194f37a543ad9cf9c8130a5d11af reported cstate change: term changed from 0 to 1, leader changed from <none> to 14a5194f37a543ad9cf9c8130a5d11af (127.3.18.2). New cstate: current_term: 1 leader_uuid: "14a5194f37a543ad9cf9c8130a5d11af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0e35e0a8fd4cec9d631c0701e0b76f" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39e67e2773bb4225a13ec3d8425a088a" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 44945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14a5194f37a543ad9cf9c8130a5d11af" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 35465 } health_report { overall_health: HEALTHY } } }
I20260928 20:19:54.691525 10578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:54.688932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43430 (local address 127.3.18.2:35465)
0928 20:19:54.689010 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:54.689012 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:54.689020 (+ 8us) server_negotiation.cc:407] Connection header received
0928 20:19:54.689074 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:54.689074 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:54.689095 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:54.689160 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:19:54.689278 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:54.689837 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:54.690727 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:54.690922 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:54.690984 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:54.691186 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:19:54.691307 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:19:54.691384 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:54.691430 (+ 46us) server_negotiation.cc:299] Negotiation successful
0928 20:19:54.691452 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 20:19:54.692775 10173 raft_consensus.cc:1261] T c965d9b923c141eb8600b166fb2df2f1 P 39e67e2773bb4225a13ec3d8425a088a [term 1 FOLLOWER]: Refusing update from remote peer 14a5194f37a543ad9cf9c8130a5d11af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:19:54.692874 10446 raft_consensus.cc:1261] T c965d9b923c141eb8600b166fb2df2f1 P 1f0e35e0a8fd4cec9d631c0701e0b76f [term 1 FOLLOWER]: Refusing update from remote peer 14a5194f37a543ad9cf9c8130a5d11af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:19:54.693109 10587 consensus_queue.cc:1035] T c965d9b923c141eb8600b166fb2df2f1 P 14a5194f37a543ad9cf9c8130a5d11af [LEADER]: Connected to new peer: Peer: permanent_uuid: "39e67e2773bb4225a13ec3d8425a088a" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 44945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:19:54.693252 10594 consensus_queue.cc:1035] T c965d9b923c141eb8600b166fb2df2f1 P 14a5194f37a543ad9cf9c8130a5d11af [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0e35e0a8fd4cec9d631c0701e0b76f" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:19:54.694145 10590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:54.694015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:42379 (local address 127.3.18.1:46501)
0928 20:19:54.694067 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:54.694101 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:42379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 20:19:54.694808 10592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:54.694475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:42379 (local address 127.3.18.3:57143)
0928 20:19:54.694718 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:54.694753 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:42379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223}
I20260928 20:19:54.696086 10588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:54.695959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:42379 (local address 127.3.18.2:40713)
0928 20:19:54.696018 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:54.696047 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:42379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 20:19:54.696136 10592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:54.696045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:42379 (local address 127.3.18.3:47767)
0928 20:19:54.696082 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:54.696105 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:42379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260928 20:19:54.697866 10590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:54.697738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:42379 (local address 127.3.18.1:47031)
0928 20:19:54.697799 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:54.697835 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:42379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 20:19:54.698457 10588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:54.698223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:42379 (local address 127.3.18.2:37761)
0928 20:19:54.698384 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:54.698425 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:42379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
I20260928 20:19:54.698479 10039 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:19:54.698482 10598 mvcc.cc:205] Tried to move back new op lower bound from 7334407351059484672 to 7334407350962016256. Current Snapshot: MvccSnapshot[applied={T|T < 7334407351059484672 or (T in {7334407351059484672})}]
I20260928 20:19:54.698576 10039 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:19:54.699913 10039 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 645a6ba8e676428585dc367ef37372c4: Sending DeleteTablet for 3 replicas of tablet c965d9b923c141eb8600b166fb2df2f1
I20260928 20:19:54.700285 10153 tablet_service.cc:1514] Processing DeleteTablet for tablet c965d9b923c141eb8600b166fb2df2f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:19:54 UTC) from {username='oryx', principal='oryx/127.3.18.60@KRBTEST.COM'} at 127.0.0.1:59932
I20260928 20:19:54.700523 10426 tablet_service.cc:1514] Processing DeleteTablet for tablet c965d9b923c141eb8600b166fb2df2f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:19:54 UTC) from {username='oryx', principal='oryx/127.3.18.60@KRBTEST.COM'} at 127.0.0.1:53482
I20260928 20:19:54.700537 10606 tablet_replica.cc:330] T c965d9b923c141eb8600b166fb2df2f1 P 39e67e2773bb4225a13ec3d8425a088a: stopping tablet replica
I20260928 20:19:54.700608 10606 raft_consensus.cc:2229] T c965d9b923c141eb8600b166fb2df2f1 P 39e67e2773bb4225a13ec3d8425a088a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:19:54.700663 10606 raft_consensus.cc:2258] T c965d9b923c141eb8600b166fb2df2f1 P 39e67e2773bb4225a13ec3d8425a088a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:19:54.700706 10290 tablet_service.cc:1514] Processing DeleteTablet for tablet c965d9b923c141eb8600b166fb2df2f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:19:54 UTC) from {username='oryx', principal='oryx/127.3.18.60@KRBTEST.COM'} at 127.0.0.1:43416
I20260928 20:19:54.700987 10606 ts_tablet_manager.cc:1905] T c965d9b923c141eb8600b166fb2df2f1 P 39e67e2773bb4225a13ec3d8425a088a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:19:54.701007 10607 tablet_replica.cc:330] T c965d9b923c141eb8600b166fb2df2f1 P 1f0e35e0a8fd4cec9d631c0701e0b76f: stopping tablet replica
I20260928 20:19:54.701063 10607 raft_consensus.cc:2229] T c965d9b923c141eb8600b166fb2df2f1 P 1f0e35e0a8fd4cec9d631c0701e0b76f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:19:54.701109 10607 raft_consensus.cc:2258] T c965d9b923c141eb8600b166fb2df2f1 P 1f0e35e0a8fd4cec9d631c0701e0b76f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:19:54.701303 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 10001
I20260928 20:19:54.702489 10607 ts_tablet_manager.cc:1905] T c965d9b923c141eb8600b166fb2df2f1 P 1f0e35e0a8fd4cec9d631c0701e0b76f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:19:54.704799 10607 ts_tablet_manager.cc:1918] T c965d9b923c141eb8600b166fb2df2f1 P 1f0e35e0a8fd4cec9d631c0701e0b76f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:19:54.704865 10607 log.cc:1196] T c965d9b923c141eb8600b166fb2df2f1 P 1f0e35e0a8fd4cec9d631c0701e0b76f: Deleting WAL directory at /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790626764250411-3144-0/minicluster-data/ts-2/wal/wals/c965d9b923c141eb8600b166fb2df2f1
I20260928 20:19:54.705163 10607 ts_tablet_manager.cc:1939] T c965d9b923c141eb8600b166fb2df2f1 P 1f0e35e0a8fd4cec9d631c0701e0b76f: Deleting consensus metadata
I20260928 20:19:54.705789 10024 catalog_manager.cc:4841] TS 1f0e35e0a8fd4cec9d631c0701e0b76f (127.3.18.3:38189): tablet c965d9b923c141eb8600b166fb2df2f1 (table test-table [id=8c9b2e8ed370455db41cd6a1b4988ee5]) successfully deleted
I20260928 20:19:54.709087 10608 tablet_replica.cc:330] T c965d9b923c141eb8600b166fb2df2f1 P 14a5194f37a543ad9cf9c8130a5d11af: stopping tablet replica
I20260928 20:19:54.709159 10561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:54.709031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.1:44945 (local address 127.0.0.1:59948)
0928 20:19:54.709084 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:54.709111 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.1:44945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 20:19:54.709185 10608 raft_consensus.cc:2229] T c965d9b923c141eb8600b166fb2df2f1 P 14a5194f37a543ad9cf9c8130a5d11af [term 1 LEADER]: Raft consensus shutting down.
I20260928 20:19:54.709290 10608 raft_consensus.cc:2258] T c965d9b923c141eb8600b166fb2df2f1 P 14a5194f37a543ad9cf9c8130a5d11af [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 20:19:54.709321 10026 catalog_manager.cc:4584] TS 39e67e2773bb4225a13ec3d8425a088a (127.3.18.1:44945): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c965d9b923c141eb8600b166fb2df2f1: Network error: Client connection negotiation failed: client connection to 127.3.18.1:44945: connect: Connection refused (error 111)
I20260928 20:19:54.709606 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 10227
I20260928 20:19:54.715309 10561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:54.715162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.2:35465 (local address 127.0.0.1:43432)
0928 20:19:54.715231 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:54.715269 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.2:35465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 20:19:54.715394 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 10363
I20260928 20:19:54.722167 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 9721
I20260928 20:19:54.727924 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 9908
2026-09-28T20:19:54Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4996 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:19:54 dist-test-slave-b83v krb5kdc[10611](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:19:54 dist-test-slave-b83v krb5kdc[10611](info): set up 2 sockets
Sep 28 20:19:54 dist-test-slave-b83v krb5kdc[10611](info): commencing operation
krb5kdc: starting...
W20260928 20:19:55.801452 3144 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Sep 28 20:19:55 dist-test-slave-b83v krb5kdc[10611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:19:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:19:55Z Disabled control of system clock
WARNING: no policy specified for oryx/127.3.18.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.18.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.18.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/krb5kdc/oryx_127.3.18.62.keytab.
Entry for principal oryx/127.3.18.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/krb5kdc/oryx_127.3.18.62.keytab.
I20260928 20:19:55.940465 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:40593
--webserver_interface=127.3.18.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/krb5kdc/oryx_127.3.18.62.keytab
--principal=oryx/127.3.18.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:43253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.18.62:40593
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:19:56.027307 10624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:19:56.027527 10624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:19:56.027549 10624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:19:56.028715 10624 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:19:56.028756 10624 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 20:19:56.028770 10624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:19:56.028784 10624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:19:56.028795 10624 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:19:56.028807 10624 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:19:56.030308 10624 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:43253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.18.62:40593
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/krb5kdc/oryx_127.3.18.62.keytab
--principal=oryx/127.3.18.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:40593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.18.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:19:56.030539 10624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:19:56.030738 10624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:19:56.036118 10629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:56.036108 10632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:56.036108 10630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:56.036504 10624 server_base.cc:1034] running on GCE node
I20260928 20:19:56.036775 10624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:19:56.037191 10624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:19:56.038414 10624 hybrid_clock.cc:648] HybridClock initialized: now 1790626796038399 us; error 36 us; skew 500 ppm
Sep 28 20:19:56 dist-test-slave-b83v krb5kdc[10611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626796, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.18.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:19:56.041522 10624 init.cc:377] Logged in from keytab as oryx/127.3.18.62@KRBTEST.COM (short username oryx)
I20260928 20:19:56.041983 10624 webserver.cc:460] Webserver started at http://127.3.18.62:38287/ using document root <none> and password file <none>
I20260928 20:19:56.042228 10624 fs_manager.cc:359] Metadata directory not provided
I20260928 20:19:56.042274 10624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:19:56.042369 10624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:19:56.043462 10624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/master-0/data/instance:
uuid: "abfa814b87a34399b12eef4a1de11f23"
format_stamp: "Formatted at 2026-09-28 20:19:56 on dist-test-slave-b83v"
server_key: "cb0acab7afabc7c3105baaaa4dd0cfdd"
server_key_iv: "9096a7a78b1504e043020db9685d6c6d"
server_key_version: "encryptionkey@0"
I20260928 20:19:56.043853 10624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/master-0/wal/instance:
uuid: "abfa814b87a34399b12eef4a1de11f23"
format_stamp: "Formatted at 2026-09-28 20:19:56 on dist-test-slave-b83v"
server_key: "cb0acab7afabc7c3105baaaa4dd0cfdd"
server_key_iv: "9096a7a78b1504e043020db9685d6c6d"
server_key_version: "encryptionkey@0"
I20260928 20:19:56.045313 10624 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:19:56.046198 10639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:56.046425 10624 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:19:56.046526 10624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/master-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/master-0/wal
uuid: "abfa814b87a34399b12eef4a1de11f23"
format_stamp: "Formatted at 2026-09-28 20:19:56 on dist-test-slave-b83v"
server_key: "cb0acab7afabc7c3105baaaa4dd0cfdd"
server_key_iv: "9096a7a78b1504e043020db9685d6c6d"
server_key_version: "encryptionkey@0"
I20260928 20:19:56.046605 10624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:19:56.069078 10624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:19:56.071395 10624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:19:56.071521 10624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:19:56.076062 10624 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.62:40593
I20260928 20:19:56.076165 10701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.62:40593 every 8 connection(s)
I20260928 20:19:56.076529 10624 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
I20260928 20:19:56.077279 10702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:56.079598 10702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abfa814b87a34399b12eef4a1de11f23: Bootstrap starting.
I20260928 20:19:56.080271 10702 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abfa814b87a34399b12eef4a1de11f23: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:56.080592 10702 log.cc:824] T 00000000000000000000000000000000 P abfa814b87a34399b12eef4a1de11f23: Log is configured to *not* fsync() on all Append() calls
I20260928 20:19:56.081312 10702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abfa814b87a34399b12eef4a1de11f23: No bootstrap required, opened a new log
I20260928 20:19:56.083387 10702 raft_consensus.cc:348] T 00000000000000000000000000000000 P abfa814b87a34399b12eef4a1de11f23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abfa814b87a34399b12eef4a1de11f23" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 40593 } }
I20260928 20:19:56.083531 10702 raft_consensus.cc:374] T 00000000000000000000000000000000 P abfa814b87a34399b12eef4a1de11f23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:56.083554 10702 raft_consensus.cc:729] T 00000000000000000000000000000000 P abfa814b87a34399b12eef4a1de11f23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abfa814b87a34399b12eef4a1de11f23, State: Initialized, Role: FOLLOWER
I20260928 20:19:56.083720 10702 consensus_queue.cc:260] T 00000000000000000000000000000000 P abfa814b87a34399b12eef4a1de11f23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abfa814b87a34399b12eef4a1de11f23" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 40593 } }
I20260928 20:19:56.083849 10702 raft_consensus.cc:388] T 00000000000000000000000000000000 P abfa814b87a34399b12eef4a1de11f23 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:19:56.083896 10702 raft_consensus.cc:482] T 00000000000000000000000000000000 P abfa814b87a34399b12eef4a1de11f23 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:19:56.083949 10702 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abfa814b87a34399b12eef4a1de11f23 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:56.084510 10702 raft_consensus.cc:504] T 00000000000000000000000000000000 P abfa814b87a34399b12eef4a1de11f23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abfa814b87a34399b12eef4a1de11f23" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 40593 } }
I20260928 20:19:56.084625 10702 leader_election.cc:302] T 00000000000000000000000000000000 P abfa814b87a34399b12eef4a1de11f23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abfa814b87a34399b12eef4a1de11f23; no voters:
I20260928 20:19:56.084803 10702 leader_election.cc:288] T 00000000000000000000000000000000 P abfa814b87a34399b12eef4a1de11f23 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:19:56.084877 10705 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abfa814b87a34399b12eef4a1de11f23 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:19:56.085093 10702 sys_catalog.cc:564] T 00000000000000000000000000000000 P abfa814b87a34399b12eef4a1de11f23 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:19:56.085104 10705 raft_consensus.cc:686] T 00000000000000000000000000000000 P abfa814b87a34399b12eef4a1de11f23 [term 1 LEADER]: Becoming Leader. State: Replica: abfa814b87a34399b12eef4a1de11f23, State: Running, Role: LEADER
I20260928 20:19:56.085302 10705 consensus_queue.cc:237] T 00000000000000000000000000000000 P abfa814b87a34399b12eef4a1de11f23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abfa814b87a34399b12eef4a1de11f23" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 40593 } }
I20260928 20:19:56.085727 10707 sys_catalog.cc:455] T 00000000000000000000000000000000 P abfa814b87a34399b12eef4a1de11f23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader abfa814b87a34399b12eef4a1de11f23. Latest consensus state: current_term: 1 leader_uuid: "abfa814b87a34399b12eef4a1de11f23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abfa814b87a34399b12eef4a1de11f23" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 40593 } } }
I20260928 20:19:56.085712 10706 sys_catalog.cc:455] T 00000000000000000000000000000000 P abfa814b87a34399b12eef4a1de11f23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "abfa814b87a34399b12eef4a1de11f23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abfa814b87a34399b12eef4a1de11f23" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 40593 } } }
I20260928 20:19:56.085807 10707 sys_catalog.cc:458] T 00000000000000000000000000000000 P abfa814b87a34399b12eef4a1de11f23 [sys.catalog]: This master's current role is: LEADER
I20260928 20:19:56.085814 10706 sys_catalog.cc:458] T 00000000000000000000000000000000 P abfa814b87a34399b12eef4a1de11f23 [sys.catalog]: This master's current role is: LEADER
I20260928 20:19:56.086088 10714 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:19:56.086331 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 10624
I20260928 20:19:56.086433 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/master-0/wal/instance
I20260928 20:19:56.086623 3144 external_mini_cluster.cc:1428] Setting key e120e09d8581ede93a71808067fae5f7
I20260928 20:19:56.086665 10714 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:19:56.088443 10714 catalog_manager.cc:1347] Generated new cluster ID: 9c2fdb6084ac424ba7a5e26b04f9892c
I20260928 20:19:56.088510 10714 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 20:19:56 dist-test-slave-b83v krb5kdc[10611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM
I20260928 20:19:56.099752 10724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:56.087328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40070 (local address 127.3.18.62:40593)
0928 20:19:56.087471 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:56.087480 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:56.087703 (+ 223us) server_negotiation.cc:407] Connection header received
0928 20:19:56.087870 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:56.087879 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:56.087941 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:56.088122 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:56.088885 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:56.090292 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:56.091424 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:56.092108 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:56.092213 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:56.095393 (+ 3180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:56.095399 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:56.095400 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:56.095414 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:56.097526 (+ 2112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:56.097941 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:56.097945 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:56.097945 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:56.098070 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:56.098275 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:56.098276 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:56.098276 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:56.098628 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:56.098776 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:56.099323 (+ 547us) server_negotiation.cc:299] Negotiation successful
0928 20:19:56.099500 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":70,"threads_started":1}
I20260928 20:19:56.109042 10714 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:19:56.109829 10714 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:19:56.115442 10714 catalog_manager.cc:5868] T 00000000000000000000000000000000 P abfa814b87a34399b12eef4a1de11f23: Generated new TSK 0
I20260928 20:19:56.115664 10714 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.3.18.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.18.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.18.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/krb5kdc/oryx_127.3.18.1.keytab.
Entry for principal oryx/127.3.18.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/krb5kdc/oryx_127.3.18.1.keytab.
I20260928 20:19:56.151950 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.1:0
--local_ip_for_outbound_sockets=127.3.18.1
--webserver_interface=127.3.18.1
--webserver_port=0
--tserver_master_addrs=127.3.18.62:40593
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/krb5kdc/oryx_127.3.18.1.keytab
--principal=oryx/127.3.18.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:43253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:19:56.240450 10728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:19:56.240743 10728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:19:56.240795 10728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:19:56.242168 10728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:19:56.242280 10728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:19:56.242337 10728 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 20:19:56.242410 10728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.1
I20260928 20:19:56.244227 10728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:43253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/krb5kdc/oryx_127.3.18.1.keytab
--principal=oryx/127.3.18.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.18.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.18.62:40593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.1
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:19:56.244568 10728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:19:56.244874 10728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:19:56.250715 10734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:56.250779 10728 server_base.cc:1034] running on GCE node
W20260928 20:19:56.250738 10736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:56.250715 10733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:56.251271 10728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:19:56.251515 10728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:19:56.252671 10728 hybrid_clock.cc:648] HybridClock initialized: now 1790626796252651 us; error 44 us; skew 500 ppm
Sep 28 20:19:56 dist-test-slave-b83v krb5kdc[10611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626796, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.18.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:19:56.255497 10728 init.cc:377] Logged in from keytab as oryx/127.3.18.1@KRBTEST.COM (short username oryx)
I20260928 20:19:56.256090 10728 webserver.cc:460] Webserver started at http://127.3.18.1:39809/ using document root <none> and password file <none>
I20260928 20:19:56.256359 10728 fs_manager.cc:359] Metadata directory not provided
I20260928 20:19:56.256408 10728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:19:56.256548 10728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:19:56.257642 10728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/ts-0/data/instance:
uuid: "74bd37a79daf47b796b54e85fbf4e212"
format_stamp: "Formatted at 2026-09-28 20:19:56 on dist-test-slave-b83v"
server_key: "00b9e9016d63cd701baadbf0ab357ded"
server_key_iv: "ffd033f983efb8259642aea987820514"
server_key_version: "encryptionkey@0"
I20260928 20:19:56.258008 10728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/ts-0/wal/instance:
uuid: "74bd37a79daf47b796b54e85fbf4e212"
format_stamp: "Formatted at 2026-09-28 20:19:56 on dist-test-slave-b83v"
server_key: "00b9e9016d63cd701baadbf0ab357ded"
server_key_iv: "ffd033f983efb8259642aea987820514"
server_key_version: "encryptionkey@0"
I20260928 20:19:56.259369 10728 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:19:56.260236 10743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:56.260402 10728 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:56.260506 10728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/ts-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/ts-0/wal
uuid: "74bd37a79daf47b796b54e85fbf4e212"
format_stamp: "Formatted at 2026-09-28 20:19:56 on dist-test-slave-b83v"
server_key: "00b9e9016d63cd701baadbf0ab357ded"
server_key_iv: "ffd033f983efb8259642aea987820514"
server_key_version: "encryptionkey@0"
I20260928 20:19:56.260633 10728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:19:56.277244 10728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:19:56.279932 10728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:19:56.280119 10728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:19:56.280699 10728 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:19:56.280769 10728 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:56.280828 10728 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:19:56.280856 10728 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 20:19:56 dist-test-slave-b83v krb5kdc[10611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626796, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.18.1@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM
I20260928 20:19:56.288718 10728 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.1:40333
I20260928 20:19:56.289119 10858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.1:40333 every 8 connection(s)
I20260928 20:19:56.289217 10728 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
I20260928 20:19:56.291844 10724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:56.281271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:46033 (local address 127.3.18.62:40593)
0928 20:19:56.281340 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:56.281344 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:56.281540 (+ 196us) server_negotiation.cc:407] Connection header received
0928 20:19:56.282156 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:56.282158 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:56.282176 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:56.282234 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:56.283383 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:56.284112 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:56.285329 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:56.285719 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:56.285786 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:56.288426 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:56.288428 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:56.288429 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:56.288440 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:56.290191 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:56.290689 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:56.290689 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:56.290690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:56.290749 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:56.291096 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:56.291098 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:56.291098 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:56.291239 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:56.291309 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:56.291625 (+ 316us) server_negotiation.cc:299] Negotiation successful
0928 20:19:56.291707 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 20:19:56.292064 10752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:56.281260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40593 (local address 127.3.18.1:46033)
0928 20:19:56.281426 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:56.281468 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:56.282029 (+ 561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:56.282382 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:56.282388 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:56.282884 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:56.283243 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:56.283254 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:56.284231 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:56.284232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:56.285227 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:56.285230 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:56.285824 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:56.285825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:56.285984 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:56.286655 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:19:56.286676 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:56.288305 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:56.290315 (+ 2010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:56.290330 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:56.290332 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:56.290604 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:56.290835 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:56.290836 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:56.290836 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:56.290965 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:56.291330 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:56.291337 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:56.291486 (+ 149us) client_negotiation.cc:769] Sending connection context
0928 20:19:56.291588 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 20:19:56.291726 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":99,"threads_started":1}
I20260928 20:19:56.292697 10859 heartbeater.cc:347] Connected to a master server at 127.3.18.62:40593
I20260928 20:19:56.292783 10859 heartbeater.cc:464] Registering TS with master...
I20260928 20:19:56.293025 10859 heartbeater.cc:511] Master 127.3.18.62:40593 requested a full tablet report, sending...
I20260928 20:19:56.293673 10656 ts_manager.cc:194] Registered new tserver with Master: 74bd37a79daf47b796b54e85fbf4e212 (127.3.18.1:40333)
I20260928 20:19:56.294662 10656 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.18.1@KRBTEST.COM'} at 127.3.18.1:46033
I20260928 20:19:56.298033 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 10728
I20260928 20:19:56.298202 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/ts-0/wal/instance
I20260928 20:19:56.298455 3144 external_mini_cluster.cc:1428] Setting key 2a93c32b4749e75a3180f1da811f57c7
I20260928 20:19:56.300215 10865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:56.293902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40593 (local address 127.3.18.1:42449)
0928 20:19:56.294054 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:56.294066 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:56.294130 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:56.294248 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:56.294249 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:56.294450 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:56.294520 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:56.294521 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:56.295324 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:56.295325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:56.296045 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:56.296048 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:56.296377 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:56.296378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:56.296426 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:56.296890 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:19:56.296895 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:56.297201 (+ 306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:56.299056 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:56.299059 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:56.299059 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:56.299248 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:56.299446 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:56.299446 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:56.299447 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:56.299494 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:56.299848 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:56.299849 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:56.299914 (+ 65us) client_negotiation.cc:769] Sending connection context
0928 20:19:56.300014 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 20:19:56.300128 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":38,"thread_start_us":86,"threads_started":1}
I20260928 20:19:56.300300 10724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:56.293990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:42449 (local address 127.3.18.62:40593)
0928 20:19:56.294040 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:56.294042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:56.294114 (+ 72us) server_negotiation.cc:407] Connection header received
0928 20:19:56.294186 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:56.294186 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:56.294194 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:56.294235 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:56.294669 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:56.295240 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:56.296120 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:56.296305 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:56.296360 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:56.297285 (+ 925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:56.297286 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:56.297286 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:56.297293 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:56.298961 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:56.299316 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:56.299317 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:56.299318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:56.299387 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:56.299553 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:56.299554 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:56.299554 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:56.299705 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:56.299802 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:56.300059 (+ 257us) server_negotiation.cc:299] Negotiation successful
0928 20:19:56.300160 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
WARNING: no policy specified for oryx/127.3.18.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.18.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.18.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/krb5kdc/oryx_127.3.18.2.keytab.
Entry for principal oryx/127.3.18.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/krb5kdc/oryx_127.3.18.2.keytab.
I20260928 20:19:56.328765 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.2:0
--local_ip_for_outbound_sockets=127.3.18.2
--webserver_interface=127.3.18.2
--webserver_port=0
--tserver_master_addrs=127.3.18.62:40593
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/krb5kdc/oryx_127.3.18.2.keytab
--principal=oryx/127.3.18.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:43253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:19:56.414641 10868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:19:56.414860 10868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:19:56.414886 10868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:19:56.416049 10868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:19:56.416088 10868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:19:56.416109 10868 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 20:19:56.416126 10868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.2
I20260928 20:19:56.417589 10868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:43253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/krb5kdc/oryx_127.3.18.2.keytab
--principal=oryx/127.3.18.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.18.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.18.62:40593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.2
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:19:56.417783 10868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:19:56.417964 10868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:19:56.423062 10873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:56.423167 10874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:56.423177 10868 server_base.cc:1034] running on GCE node
W20260928 20:19:56.423058 10876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:56.423625 10868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:19:56.423913 10868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:19:56.425092 10868 hybrid_clock.cc:648] HybridClock initialized: now 1790626796425077 us; error 32 us; skew 500 ppm
Sep 28 20:19:56 dist-test-slave-b83v krb5kdc[10611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626796, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.18.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:19:56.427656 10868 init.cc:377] Logged in from keytab as oryx/127.3.18.2@KRBTEST.COM (short username oryx)
I20260928 20:19:56.428179 10868 webserver.cc:460] Webserver started at http://127.3.18.2:42979/ using document root <none> and password file <none>
I20260928 20:19:56.428431 10868 fs_manager.cc:359] Metadata directory not provided
I20260928 20:19:56.428510 10868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:19:56.428661 10868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:19:56.429738 10868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/ts-1/data/instance:
uuid: "548a75d717894f7e8066c6e9ddbb15c7"
format_stamp: "Formatted at 2026-09-28 20:19:56 on dist-test-slave-b83v"
server_key: "3f54975f2fe3c0b41ce9e175e18c1eda"
server_key_iv: "12f3a78a5b9d0d1610a3a276dc122abe"
server_key_version: "encryptionkey@0"
I20260928 20:19:56.430114 10868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/ts-1/wal/instance:
uuid: "548a75d717894f7e8066c6e9ddbb15c7"
format_stamp: "Formatted at 2026-09-28 20:19:56 on dist-test-slave-b83v"
server_key: "3f54975f2fe3c0b41ce9e175e18c1eda"
server_key_iv: "12f3a78a5b9d0d1610a3a276dc122abe"
server_key_version: "encryptionkey@0"
I20260928 20:19:56.431573 10868 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 20:19:56.432545 10883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:56.432716 10868 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:19:56.432806 10868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/ts-1/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/ts-1/wal
uuid: "548a75d717894f7e8066c6e9ddbb15c7"
format_stamp: "Formatted at 2026-09-28 20:19:56 on dist-test-slave-b83v"
server_key: "3f54975f2fe3c0b41ce9e175e18c1eda"
server_key_iv: "12f3a78a5b9d0d1610a3a276dc122abe"
server_key_version: "encryptionkey@0"
I20260928 20:19:56.432859 10868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:19:56.447086 10868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:19:56.449378 10868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:19:56.449534 10868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:19:56.450176 10868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:19:56.450227 10868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:56.450259 10868 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:19:56.450280 10868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 20:19:56 dist-test-slave-b83v krb5kdc[10611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626796, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.18.2@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM
I20260928 20:19:56.457986 10868 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.2:45543
I20260928 20:19:56.458410 10868 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
I20260928 20:19:56.458371 10998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.2:45543 every 8 connection(s)
I20260928 20:19:56.461414 10724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:56.450603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:42421 (local address 127.3.18.62:40593)
0928 20:19:56.450688 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:56.450690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:56.450902 (+ 212us) server_negotiation.cc:407] Connection header received
0928 20:19:56.451472 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:56.451473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:56.451487 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:56.451535 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:56.452443 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:56.452956 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:56.454024 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:56.454233 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:56.454324 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:56.458022 (+ 3698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:56.458024 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:56.458025 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:56.458035 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:56.459936 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:56.460395 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:56.460396 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:56.460397 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:56.460464 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:56.460734 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:56.460734 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:56.460735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:56.460863 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:56.460940 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:56.461236 (+ 296us) server_negotiation.cc:299] Negotiation successful
0928 20:19:56.461322 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 20:19:56.461637 10892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:56.450617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40593 (local address 127.3.18.2:42421)
0928 20:19:56.450768 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:56.450811 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:56.451350 (+ 539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:56.451712 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:56.451716 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:56.452122 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:56.452337 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:56.452343 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:56.453071 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:56.453072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:56.453922 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:56.453924 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:56.454339 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:56.454340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:56.454733 (+ 393us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:56.455434 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:19:56.455444 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:56.457888 (+ 2444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:56.460041 (+ 2153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:56.460048 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:56.460050 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:56.460291 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:56.460534 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:56.460535 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:56.460536 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:56.460654 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:56.460930 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:56.460936 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:56.461089 (+ 153us) client_negotiation.cc:769] Sending connection context
0928 20:19:56.461199 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 20:19:56.461374 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":73,"threads_started":1}
I20260928 20:19:56.462145 10999 heartbeater.cc:347] Connected to a master server at 127.3.18.62:40593
I20260928 20:19:56.462258 10999 heartbeater.cc:464] Registering TS with master...
I20260928 20:19:56.462463 10999 heartbeater.cc:511] Master 127.3.18.62:40593 requested a full tablet report, sending...
I20260928 20:19:56.462960 10656 ts_manager.cc:194] Registered new tserver with Master: 548a75d717894f7e8066c6e9ddbb15c7 (127.3.18.2:45543)
I20260928 20:19:56.463431 10656 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.18.2@KRBTEST.COM'} at 127.3.18.2:42421
I20260928 20:19:56.464860 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 10868
I20260928 20:19:56.464987 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/ts-1/wal/instance
I20260928 20:19:56.465194 3144 external_mini_cluster.cc:1428] Setting key 157ebd7505c9ea9e36c3cb5fcba634f0
I20260928 20:19:56.470487 10724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:56.463293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:45165 (local address 127.3.18.62:40593)
0928 20:19:56.463367 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:56.463370 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:56.463505 (+ 135us) server_negotiation.cc:407] Connection header received
0928 20:19:56.463599 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:56.463601 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:56.463612 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:56.463677 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:56.464109 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:56.464741 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:56.465703 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:56.466008 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:56.466116 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:56.467335 (+ 1219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:56.467336 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:56.467336 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:56.467344 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:56.469072 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:56.469506 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:56.469507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:56.469508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:56.469575 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:56.469802 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:56.469803 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:56.469804 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:56.469976 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:56.470083 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:56.470277 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 20:19:56.470375 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 20:19:56.470500 11005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:56.463232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40593 (local address 127.3.18.2:45165)
0928 20:19:56.463425 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:56.463442 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:56.463520 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:56.463680 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:56.463682 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:56.463925 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:56.464004 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:56.464007 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:56.464860 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:56.464861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:56.465624 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:56.465626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:56.466131 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:56.466132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:56.466207 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:56.466735 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:19:56.466742 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:56.467228 (+ 486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:56.469199 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:56.469201 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:56.469201 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:56.469408 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:56.469652 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:56.469653 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:56.469654 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:56.469730 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:56.470075 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:56.470076 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:56.470133 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 20:19:56.470248 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 20:19:56.470376 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":90,"threads_started":1}
WARNING: no policy specified for oryx/127.3.18.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.18.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.18.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/krb5kdc/oryx_127.3.18.3.keytab.
Entry for principal oryx/127.3.18.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/krb5kdc/oryx_127.3.18.3.keytab.
I20260928 20:19:56.494685 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.3:0
--local_ip_for_outbound_sockets=127.3.18.3
--webserver_interface=127.3.18.3
--webserver_port=0
--tserver_master_addrs=127.3.18.62:40593
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/krb5kdc/oryx_127.3.18.3.keytab
--principal=oryx/127.3.18.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:43253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:19:56.578090 11008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:19:56.578299 11008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:19:56.578323 11008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:19:56.579483 11008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:19:56.579530 11008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:19:56.579551 11008 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 20:19:56.579568 11008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.3
I20260928 20:19:56.581086 11008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:43253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/krb5kdc/oryx_127.3.18.3.keytab
--principal=oryx/127.3.18.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.18.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.18.62:40593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.3
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:19:56.581352 11008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:19:56.581604 11008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:19:56.587186 11016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:56.587191 11014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:56.587244 11008 server_base.cc:1034] running on GCE node
W20260928 20:19:56.587212 11013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:56.587687 11008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:19:56.588001 11008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:19:56.589190 11008 hybrid_clock.cc:648] HybridClock initialized: now 1790626796589170 us; error 40 us; skew 500 ppm
Sep 28 20:19:56 dist-test-slave-b83v krb5kdc[10611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626796, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.18.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:19:56.591958 11008 init.cc:377] Logged in from keytab as oryx/127.3.18.3@KRBTEST.COM (short username oryx)
I20260928 20:19:56.592443 11008 webserver.cc:460] Webserver started at http://127.3.18.3:35063/ using document root <none> and password file <none>
I20260928 20:19:56.592664 11008 fs_manager.cc:359] Metadata directory not provided
I20260928 20:19:56.592706 11008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:19:56.592799 11008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:19:56.593930 11008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/ts-2/data/instance:
uuid: "f29fc7bde86e4c0faed6562bce8a47f4"
format_stamp: "Formatted at 2026-09-28 20:19:56 on dist-test-slave-b83v"
server_key: "250c8fb4d115dc3d431f98d46b262ede"
server_key_iv: "3985475407e1feae89a488179359fd0e"
server_key_version: "encryptionkey@0"
I20260928 20:19:56.594249 11008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/ts-2/wal/instance:
uuid: "f29fc7bde86e4c0faed6562bce8a47f4"
format_stamp: "Formatted at 2026-09-28 20:19:56 on dist-test-slave-b83v"
server_key: "250c8fb4d115dc3d431f98d46b262ede"
server_key_iv: "3985475407e1feae89a488179359fd0e"
server_key_version: "encryptionkey@0"
I20260928 20:19:56.595628 11008 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 20:19:56.596525 11023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:56.596688 11008 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:19:56.596760 11008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/ts-2/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/ts-2/wal
uuid: "f29fc7bde86e4c0faed6562bce8a47f4"
format_stamp: "Formatted at 2026-09-28 20:19:56 on dist-test-slave-b83v"
server_key: "250c8fb4d115dc3d431f98d46b262ede"
server_key_iv: "3985475407e1feae89a488179359fd0e"
server_key_version: "encryptionkey@0"
I20260928 20:19:56.596815 11008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:19:56.608063 11008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:19:56.610543 11008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:19:56.610670 11008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:19:56.611147 11008 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:19:56.611186 11008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:56.611255 11008 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:19:56.611294 11008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:56.618909 11008 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.3:36917
I20260928 20:19:56.619001 11138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.3:36917 every 8 connection(s)
Sep 28 20:19:56 dist-test-slave-b83v krb5kdc[10611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626796, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.18.3@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM
I20260928 20:19:56.619371 11008 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
I20260928 20:19:56.619946 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 11008
I20260928 20:19:56.620092 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/ts-2/wal/instance
I20260928 20:19:56.620283 3144 external_mini_cluster.cc:1428] Setting key 0f26a59efb3ff6176935b2fe410c04f4
I20260928 20:19:56.623030 10724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:56.611709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:40701 (local address 127.3.18.62:40593)
0928 20:19:56.611805 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:56.611808 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:56.612079 (+ 271us) server_negotiation.cc:407] Connection header received
0928 20:19:56.612830 (+ 751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:56.612831 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:56.612842 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:56.612885 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:56.613924 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:56.614791 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:56.616681 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:56.616988 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:56.617049 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:56.620136 (+ 3087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:56.620138 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:56.620138 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:56.620147 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:56.621773 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:56.622206 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:56.622207 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:56.622208 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:56.622263 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:56.622478 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:56.622479 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:56.622479 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:56.622591 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:56.622640 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:56.622896 (+ 256us) server_negotiation.cc:299] Negotiation successful
0928 20:19:56.622964 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 20:19:56.623173 11032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:56.611684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40593 (local address 127.3.18.3:40701)
0928 20:19:56.611899 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:56.611976 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:56.612721 (+ 745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:56.612990 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:56.612994 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:56.613440 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:56.613785 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:56.613794 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:56.614908 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:56.614910 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:56.616563 (+ 1653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:56.616567 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:56.617095 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:56.617096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:56.617269 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:56.617901 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:19:56.617911 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:56.619977 (+ 2066us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:56.621853 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:56.621860 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:56.621862 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:56.622127 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:56.622320 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:56.622320 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:56.622320 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:56.622418 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:56.622646 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:56.622650 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:56.622770 (+ 120us) client_negotiation.cc:769] Sending connection context
0928 20:19:56.622849 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 20:19:56.622977 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":116,"threads_started":1}
I20260928 20:19:56.623728 11139 heartbeater.cc:347] Connected to a master server at 127.3.18.62:40593
I20260928 20:19:56.623862 11139 heartbeater.cc:464] Registering TS with master...
I20260928 20:19:56.624063 11139 heartbeater.cc:511] Master 127.3.18.62:40593 requested a full tablet report, sending...
I20260928 20:19:56.624471 10655 ts_manager.cc:194] Registered new tserver with Master: f29fc7bde86e4c0faed6562bce8a47f4 (127.3.18.3:36917)
I20260928 20:19:56.624965 10655 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.18.3@KRBTEST.COM'} at 127.3.18.3:40701
I20260928 20:19:56.630493 10724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:56.624977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:55227 (local address 127.3.18.62:40593)
0928 20:19:56.625021 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:56.625023 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:56.625091 (+ 68us) server_negotiation.cc:407] Connection header received
0928 20:19:56.625160 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:56.625161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:56.625172 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:56.625240 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:56.625586 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:56.626145 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:56.626940 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:56.627134 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:56.627185 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:56.628250 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:56.628251 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:56.628251 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:56.628257 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:56.629547 (+ 1290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:56.629882 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:56.629883 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:56.629883 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:56.629932 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:56.630111 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:56.630112 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:56.630112 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:56.630210 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:56.630261 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:56.630372 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 20:19:56.630428 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 20:19:56.630609 11145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:56.624888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40593 (local address 127.3.18.3:55227)
0928 20:19:56.625029 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:56.625042 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:56.625106 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:56.625252 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:56.625253 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:56.625426 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:56.625491 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:56.625492 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:56.626269 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:56.626270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:56.626858 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:56.626860 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:56.627206 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:56.627207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:56.627265 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:56.627842 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:19:56.627848 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:56.628166 (+ 318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:56.629641 (+ 1475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:56.629643 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:56.629643 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:56.629816 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:56.629992 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:56.629993 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:56.629994 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:56.630057 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:56.630267 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:56.630268 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:56.630305 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 20:19:56.630356 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 20:19:56.630438 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":84,"threads_started":1}
I20260928 20:19:56.632146 3144 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 20:19:56 dist-test-slave-b83v krb5kdc[10611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626796, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 20:19:56 dist-test-slave-b83v krb5kdc[10611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:19:56 dist-test-slave-b83v krb5kdc[10611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 20:19:56.655155 10724 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:19:56.648852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40074 (local address 127.3.18.62:40593)
0928 20:19:56.648939 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:56.648942 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:56.648961 (+ 19us) server_negotiation.cc:407] Connection header received
0928 20:19:56.648995 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:56.648996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:56.649006 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:56.649069 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:56.649797 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:56.650393 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:56.651271 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:56.651461 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:56.651521 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:56.655038 (+ 3517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40074: BlockingRecv error: recv got EOF from 127.0.0.1:40074 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 28 20:19:56 dist-test-slave-b83v krb5kdc[10611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626796, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM
I20260928 20:19:56.663723 10724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:56.656144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40080 (local address 127.3.18.62:40593)
0928 20:19:56.656214 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:56.656216 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:56.656292 (+ 76us) server_negotiation.cc:407] Connection header received
0928 20:19:56.656387 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:56.656388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:56.656398 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:56.656457 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:56.656961 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:56.657458 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:56.658182 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:56.658385 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:56.658474 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:56.660907 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:56.660909 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:56.660910 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:56.660919 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:56.662346 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:56.662758 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:56.662760 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:56.662761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:56.662827 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:56.663068 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:56.663070 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:56.663070 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:56.663250 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:56.663364 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:56.663513 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 20:19:56.663611 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 20:19:56.666041 10655 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40080:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 20:19:56.667687 10655 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 20:19:56.674449 11162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:56.670300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.3:36917 (local address 127.0.0.1:44328)
0928 20:19:56.670451 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:56.670569 (+ 118us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:56.670637 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:56.670841 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:56.670844 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:56.670846 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:19:56.670956 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:56.670959 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:56.672045 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:56.672046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:56.673060 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:56.673062 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:56.674119 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:56.674120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:56.674194 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:56.674204 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 20:19:56.674293 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 20:19:56.674352 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":49,"threads_started":1}
I20260928 20:19:56.674454 11165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:56.670484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44328 (local address 127.3.18.3:36917)
0928 20:19:56.670645 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:56.670651 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:56.670665 (+ 14us) server_negotiation.cc:407] Connection header received
0928 20:19:56.670715 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:56.670718 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:56.670735 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:56.670817 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:19:56.671055 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:56.671942 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:56.673177 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:56.673942 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:56.674014 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:56.674090 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:56.674311 (+ 221us) server_negotiation.cc:299] Negotiation successful
0928 20:19:56.674352 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":88,"threads_started":1}
I20260928 20:19:56.675034 11164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:56.670496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.1:40333 (local address 127.0.0.1:44584)
0928 20:19:56.670935 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:56.670953 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:56.671031 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:56.671374 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:56.671375 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:56.671377 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:19:56.671407 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:56.671408 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:56.672777 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:56.672779 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:56.673701 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:56.673704 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:56.674779 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:56.674780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:56.674845 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:56.674851 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 20:19:56.674899 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 20:19:56.674944 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":62,"threads_started":1}
I20260928 20:19:56.675203 11161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:56.670306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.2:45543 (local address 127.0.0.1:41414)
0928 20:19:56.670660 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:56.670671 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:56.670727 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:56.671324 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:56.671325 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:56.671327 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:19:56.671384 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:56.671386 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:56.672766 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:56.672767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:56.673875 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:56.673878 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:56.674951 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:56.674952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:56.675020 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:56.675025 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 20:19:56.675095 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 20:19:56.675137 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":27,"thread_start_us":85,"threads_started":1}
I20260928 20:19:56.675266 11163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:56.670403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41414 (local address 127.3.18.2:45543)
0928 20:19:56.671160 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:56.671165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:56.671174 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:19:56.671210 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:56.671212 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:56.671230 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:56.671463 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:19:56.671528 (+ 65us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:56.672663 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:56.673969 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:56.674823 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:56.674927 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:56.675011 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:56.675115 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 20:19:56.675168 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":718,"thread_start_us":103,"threads_started":1}
I20260928 20:19:56.675511 11073 tablet_service.cc:1467] Processing CreateTablet for tablet f5d74938fd7d47b3b4f3882b9257644e (DEFAULT_TABLE table=test-table [id=7e0b8aa227544de69188354c9278aab8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:19:56.675789 11166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:56.670697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44584 (local address 127.3.18.1:40333)
0928 20:19:56.670890 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:56.670894 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:56.671024 (+ 130us) server_negotiation.cc:407] Connection header received
0928 20:19:56.671100 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:56.671103 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:56.671201 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:56.671283 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:19:56.671483 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:56.672653 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:56.673798 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:56.674589 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:56.674663 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:56.674742 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:56.675217 (+ 475us) server_negotiation.cc:299] Negotiation successful
0928 20:19:56.675661 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":111,"threads_started":1}
I20260928 20:19:56.676003 11073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5d74938fd7d47b3b4f3882b9257644e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:56.675927 10933 tablet_service.cc:1467] Processing CreateTablet for tablet f5d74938fd7d47b3b4f3882b9257644e (DEFAULT_TABLE table=test-table [id=7e0b8aa227544de69188354c9278aab8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:19:56.676275 10933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5d74938fd7d47b3b4f3882b9257644e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:56.679162 11167 tablet_bootstrap.cc:492] T f5d74938fd7d47b3b4f3882b9257644e P f29fc7bde86e4c0faed6562bce8a47f4: Bootstrap starting.
I20260928 20:19:56.679569 11168 tablet_bootstrap.cc:492] T f5d74938fd7d47b3b4f3882b9257644e P 548a75d717894f7e8066c6e9ddbb15c7: Bootstrap starting.
I20260928 20:19:56.679760 10793 tablet_service.cc:1467] Processing CreateTablet for tablet f5d74938fd7d47b3b4f3882b9257644e (DEFAULT_TABLE table=test-table [id=7e0b8aa227544de69188354c9278aab8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:19:56.679997 11167 tablet_bootstrap.cc:654] T f5d74938fd7d47b3b4f3882b9257644e P f29fc7bde86e4c0faed6562bce8a47f4: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:56.680197 10793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5d74938fd7d47b3b4f3882b9257644e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:56.680258 11168 tablet_bootstrap.cc:654] T f5d74938fd7d47b3b4f3882b9257644e P 548a75d717894f7e8066c6e9ddbb15c7: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:56.680378 11167 log.cc:824] T f5d74938fd7d47b3b4f3882b9257644e P f29fc7bde86e4c0faed6562bce8a47f4: Log is configured to *not* fsync() on all Append() calls
I20260928 20:19:56.680579 11168 log.cc:824] T f5d74938fd7d47b3b4f3882b9257644e P 548a75d717894f7e8066c6e9ddbb15c7: Log is configured to *not* fsync() on all Append() calls
I20260928 20:19:56.681262 11167 tablet_bootstrap.cc:492] T f5d74938fd7d47b3b4f3882b9257644e P f29fc7bde86e4c0faed6562bce8a47f4: No bootstrap required, opened a new log
I20260928 20:19:56.681352 11167 ts_tablet_manager.cc:1397] T f5d74938fd7d47b3b4f3882b9257644e P f29fc7bde86e4c0faed6562bce8a47f4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 20:19:56.681385 11168 tablet_bootstrap.cc:492] T f5d74938fd7d47b3b4f3882b9257644e P 548a75d717894f7e8066c6e9ddbb15c7: No bootstrap required, opened a new log
I20260928 20:19:56.681461 11168 ts_tablet_manager.cc:1397] T f5d74938fd7d47b3b4f3882b9257644e P 548a75d717894f7e8066c6e9ddbb15c7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 20:19:56.682978 11167 raft_consensus.cc:348] T f5d74938fd7d47b3b4f3882b9257644e P f29fc7bde86e4c0faed6562bce8a47f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36917 } } peers { permanent_uuid: "548a75d717894f7e8066c6e9ddbb15c7" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45543 } } peers { permanent_uuid: "74bd37a79daf47b796b54e85fbf4e212" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40333 } }
I20260928 20:19:56.683157 11167 raft_consensus.cc:374] T f5d74938fd7d47b3b4f3882b9257644e P f29fc7bde86e4c0faed6562bce8a47f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:56.683202 11167 raft_consensus.cc:729] T f5d74938fd7d47b3b4f3882b9257644e P f29fc7bde86e4c0faed6562bce8a47f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f29fc7bde86e4c0faed6562bce8a47f4, State: Initialized, Role: FOLLOWER
I20260928 20:19:56.683320 11167 consensus_queue.cc:260] T f5d74938fd7d47b3b4f3882b9257644e P f29fc7bde86e4c0faed6562bce8a47f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36917 } } peers { permanent_uuid: "548a75d717894f7e8066c6e9ddbb15c7" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45543 } } peers { permanent_uuid: "74bd37a79daf47b796b54e85fbf4e212" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40333 } }
I20260928 20:19:56.683640 11139 heartbeater.cc:503] Master 127.3.18.62:40593 was elected leader, sending a full tablet report...
I20260928 20:19:56.683650 11168 raft_consensus.cc:348] T f5d74938fd7d47b3b4f3882b9257644e P 548a75d717894f7e8066c6e9ddbb15c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36917 } } peers { permanent_uuid: "548a75d717894f7e8066c6e9ddbb15c7" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45543 } } peers { permanent_uuid: "74bd37a79daf47b796b54e85fbf4e212" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40333 } }
I20260928 20:19:56.683637 11167 ts_tablet_manager.cc:1428] T f5d74938fd7d47b3b4f3882b9257644e P f29fc7bde86e4c0faed6562bce8a47f4: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 20:19:56.683847 11168 raft_consensus.cc:374] T f5d74938fd7d47b3b4f3882b9257644e P 548a75d717894f7e8066c6e9ddbb15c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:56.683892 11168 raft_consensus.cc:729] T f5d74938fd7d47b3b4f3882b9257644e P 548a75d717894f7e8066c6e9ddbb15c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548a75d717894f7e8066c6e9ddbb15c7, State: Initialized, Role: FOLLOWER
I20260928 20:19:56.684011 11168 consensus_queue.cc:260] T f5d74938fd7d47b3b4f3882b9257644e P 548a75d717894f7e8066c6e9ddbb15c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36917 } } peers { permanent_uuid: "548a75d717894f7e8066c6e9ddbb15c7" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45543 } } peers { permanent_uuid: "74bd37a79daf47b796b54e85fbf4e212" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40333 } }
I20260928 20:19:56.684293 11168 ts_tablet_manager.cc:1428] T f5d74938fd7d47b3b4f3882b9257644e P 548a75d717894f7e8066c6e9ddbb15c7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 20:19:56.684473 11173 tablet_bootstrap.cc:492] T f5d74938fd7d47b3b4f3882b9257644e P 74bd37a79daf47b796b54e85fbf4e212: Bootstrap starting.
I20260928 20:19:56.684653 10999 heartbeater.cc:503] Master 127.3.18.62:40593 was elected leader, sending a full tablet report...
I20260928 20:19:56.685490 11173 tablet_bootstrap.cc:654] T f5d74938fd7d47b3b4f3882b9257644e P 74bd37a79daf47b796b54e85fbf4e212: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:56.685846 11173 log.cc:824] T f5d74938fd7d47b3b4f3882b9257644e P 74bd37a79daf47b796b54e85fbf4e212: Log is configured to *not* fsync() on all Append() calls
I20260928 20:19:56.686685 11173 tablet_bootstrap.cc:492] T f5d74938fd7d47b3b4f3882b9257644e P 74bd37a79daf47b796b54e85fbf4e212: No bootstrap required, opened a new log
I20260928 20:19:56.686774 11173 ts_tablet_manager.cc:1397] T f5d74938fd7d47b3b4f3882b9257644e P 74bd37a79daf47b796b54e85fbf4e212: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 20:19:56.688262 11173 raft_consensus.cc:348] T f5d74938fd7d47b3b4f3882b9257644e P 74bd37a79daf47b796b54e85fbf4e212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36917 } } peers { permanent_uuid: "548a75d717894f7e8066c6e9ddbb15c7" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45543 } } peers { permanent_uuid: "74bd37a79daf47b796b54e85fbf4e212" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40333 } }
I20260928 20:19:56.688421 11173 raft_consensus.cc:374] T f5d74938fd7d47b3b4f3882b9257644e P 74bd37a79daf47b796b54e85fbf4e212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:56.688470 11173 raft_consensus.cc:729] T f5d74938fd7d47b3b4f3882b9257644e P 74bd37a79daf47b796b54e85fbf4e212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74bd37a79daf47b796b54e85fbf4e212, State: Initialized, Role: FOLLOWER
I20260928 20:19:56.688627 11173 consensus_queue.cc:260] T f5d74938fd7d47b3b4f3882b9257644e P 74bd37a79daf47b796b54e85fbf4e212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36917 } } peers { permanent_uuid: "548a75d717894f7e8066c6e9ddbb15c7" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45543 } } peers { permanent_uuid: "74bd37a79daf47b796b54e85fbf4e212" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40333 } }
I20260928 20:19:56.688892 11173 ts_tablet_manager.cc:1428] T f5d74938fd7d47b3b4f3882b9257644e P 74bd37a79daf47b796b54e85fbf4e212: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 20:19:56.688952 10859 heartbeater.cc:503] Master 127.3.18.62:40593 was elected leader, sending a full tablet report...
W20260928 20:19:56.709030 11000 tablet.cc:2369] T f5d74938fd7d47b3b4f3882b9257644e P 548a75d717894f7e8066c6e9ddbb15c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 20:19:56.790159 10860 tablet.cc:2369] T f5d74938fd7d47b3b4f3882b9257644e P 74bd37a79daf47b796b54e85fbf4e212: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 20:19:56.870785 11140 tablet.cc:2369] T f5d74938fd7d47b3b4f3882b9257644e P f29fc7bde86e4c0faed6562bce8a47f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 20:19:56.879879 11172 raft_consensus.cc:482] T f5d74938fd7d47b3b4f3882b9257644e P 548a75d717894f7e8066c6e9ddbb15c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:19:56.880020 11172 raft_consensus.cc:504] T f5d74938fd7d47b3b4f3882b9257644e P 548a75d717894f7e8066c6e9ddbb15c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36917 } } peers { permanent_uuid: "548a75d717894f7e8066c6e9ddbb15c7" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45543 } } peers { permanent_uuid: "74bd37a79daf47b796b54e85fbf4e212" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40333 } }
I20260928 20:19:56.880375 11172 leader_election.cc:288] T f5d74938fd7d47b3b4f3882b9257644e P 548a75d717894f7e8066c6e9ddbb15c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f29fc7bde86e4c0faed6562bce8a47f4 (127.3.18.3:36917), 74bd37a79daf47b796b54e85fbf4e212 (127.3.18.1:40333)
I20260928 20:19:56.884500 11165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:56.880771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:46761 (local address 127.3.18.3:36917)
0928 20:19:56.880815 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:56.880817 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:56.880832 (+ 15us) server_negotiation.cc:407] Connection header received
0928 20:19:56.880856 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:56.880857 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:56.880866 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:56.880911 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:19:56.881146 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:56.881936 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:56.883459 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:56.884056 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:56.884142 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:56.884155 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:56.884382 (+ 227us) server_negotiation.cc:299] Negotiation successful
0928 20:19:56.884422 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 20:19:56.884518 11176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:56.880703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.1:40333 (local address 127.3.18.2:60575)
0928 20:19:56.880878 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:56.880916 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:56.881016 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:56.881247 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:56.881247 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:56.881249 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:19:56.881309 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:56.881311 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:56.882160 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:56.882161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:56.883342 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:56.883345 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:56.884190 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:56.884191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:56.884289 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:56.884296 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 20:19:56.884366 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 20:19:56.884411 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":4,"thread_start_us":126,"threads_started":1}
I20260928 20:19:56.884553 11166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:56.880748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:60575 (local address 127.3.18.1:40333)
0928 20:19:56.880799 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:56.880802 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:56.880992 (+ 190us) server_negotiation.cc:407] Connection header received
0928 20:19:56.881149 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:56.881151 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:56.881165 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:56.881238 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:19:56.881384 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:56.882093 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:56.883458 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:56.884096 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:56.884162 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:56.884171 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:56.884382 (+ 211us) server_negotiation.cc:299] Negotiation successful
0928 20:19:56.884422 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 20:19:56.884815 10892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:56.880582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.3:36917 (local address 127.3.18.2:46761)
0928 20:19:56.880643 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:56.880658 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:56.880713 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:56.880953 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:56.880954 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:56.880956 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:19:56.881033 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:56.881035 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:56.882034 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:56.882034 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:56.883342 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:56.883345 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:56.884190 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:56.884191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:56.884289 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:56.884295 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 20:19:56.884376 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 20:19:56.884411 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":37}
I20260928 20:19:56.885013 10813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5d74938fd7d47b3b4f3882b9257644e" candidate_uuid: "548a75d717894f7e8066c6e9ddbb15c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74bd37a79daf47b796b54e85fbf4e212" is_pre_election: true
I20260928 20:19:56.885232 10813 raft_consensus.cc:2454] T f5d74938fd7d47b3b4f3882b9257644e P 74bd37a79daf47b796b54e85fbf4e212 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548a75d717894f7e8066c6e9ddbb15c7 in term 0.
I20260928 20:19:56.885211 11093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5d74938fd7d47b3b4f3882b9257644e" candidate_uuid: "548a75d717894f7e8066c6e9ddbb15c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" is_pre_election: true
I20260928 20:19:56.885341 11093 raft_consensus.cc:2454] T f5d74938fd7d47b3b4f3882b9257644e P f29fc7bde86e4c0faed6562bce8a47f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548a75d717894f7e8066c6e9ddbb15c7 in term 0.
I20260928 20:19:56.885597 10886 leader_election.cc:302] T f5d74938fd7d47b3b4f3882b9257644e P 548a75d717894f7e8066c6e9ddbb15c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548a75d717894f7e8066c6e9ddbb15c7, 74bd37a79daf47b796b54e85fbf4e212; no voters:
I20260928 20:19:56.885798 11172 raft_consensus.cc:2781] T f5d74938fd7d47b3b4f3882b9257644e P 548a75d717894f7e8066c6e9ddbb15c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 20:19:56.885885 11172 raft_consensus.cc:482] T f5d74938fd7d47b3b4f3882b9257644e P 548a75d717894f7e8066c6e9ddbb15c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 20:19:56.885912 11172 raft_consensus.cc:3037] T f5d74938fd7d47b3b4f3882b9257644e P 548a75d717894f7e8066c6e9ddbb15c7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:56.887185 11172 raft_consensus.cc:504] T f5d74938fd7d47b3b4f3882b9257644e P 548a75d717894f7e8066c6e9ddbb15c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36917 } } peers { permanent_uuid: "548a75d717894f7e8066c6e9ddbb15c7" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45543 } } peers { permanent_uuid: "74bd37a79daf47b796b54e85fbf4e212" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40333 } }
I20260928 20:19:56.887362 11172 leader_election.cc:288] T f5d74938fd7d47b3b4f3882b9257644e P 548a75d717894f7e8066c6e9ddbb15c7 [CANDIDATE]: Term 1 election: Requested vote from peers f29fc7bde86e4c0faed6562bce8a47f4 (127.3.18.3:36917), 74bd37a79daf47b796b54e85fbf4e212 (127.3.18.1:40333)
I20260928 20:19:56.887568 11093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5d74938fd7d47b3b4f3882b9257644e" candidate_uuid: "548a75d717894f7e8066c6e9ddbb15c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f29fc7bde86e4c0faed6562bce8a47f4"
I20260928 20:19:56.887665 11093 raft_consensus.cc:3037] T f5d74938fd7d47b3b4f3882b9257644e P f29fc7bde86e4c0faed6562bce8a47f4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:56.887722 10813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5d74938fd7d47b3b4f3882b9257644e" candidate_uuid: "548a75d717894f7e8066c6e9ddbb15c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74bd37a79daf47b796b54e85fbf4e212"
I20260928 20:19:56.887852 10813 raft_consensus.cc:3037] T f5d74938fd7d47b3b4f3882b9257644e P 74bd37a79daf47b796b54e85fbf4e212 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:56.888475 11093 raft_consensus.cc:2454] T f5d74938fd7d47b3b4f3882b9257644e P f29fc7bde86e4c0faed6562bce8a47f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548a75d717894f7e8066c6e9ddbb15c7 in term 1.
I20260928 20:19:56.888669 10813 raft_consensus.cc:2454] T f5d74938fd7d47b3b4f3882b9257644e P 74bd37a79daf47b796b54e85fbf4e212 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548a75d717894f7e8066c6e9ddbb15c7 in term 1.
I20260928 20:19:56.888674 10885 leader_election.cc:302] T f5d74938fd7d47b3b4f3882b9257644e P 548a75d717894f7e8066c6e9ddbb15c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548a75d717894f7e8066c6e9ddbb15c7, f29fc7bde86e4c0faed6562bce8a47f4; no voters:
I20260928 20:19:56.888872 11172 raft_consensus.cc:2781] T f5d74938fd7d47b3b4f3882b9257644e P 548a75d717894f7e8066c6e9ddbb15c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:19:56.889011 11172 raft_consensus.cc:686] T f5d74938fd7d47b3b4f3882b9257644e P 548a75d717894f7e8066c6e9ddbb15c7 [term 1 LEADER]: Becoming Leader. State: Replica: 548a75d717894f7e8066c6e9ddbb15c7, State: Running, Role: LEADER
I20260928 20:19:56.889114 11172 consensus_queue.cc:237] T f5d74938fd7d47b3b4f3882b9257644e P 548a75d717894f7e8066c6e9ddbb15c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36917 } } peers { permanent_uuid: "548a75d717894f7e8066c6e9ddbb15c7" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45543 } } peers { permanent_uuid: "74bd37a79daf47b796b54e85fbf4e212" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40333 } }
I20260928 20:19:56.890194 10655 catalog_manager.cc:5495] T f5d74938fd7d47b3b4f3882b9257644e P 548a75d717894f7e8066c6e9ddbb15c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 548a75d717894f7e8066c6e9ddbb15c7 (127.3.18.2). New cstate: current_term: 1 leader_uuid: "548a75d717894f7e8066c6e9ddbb15c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "548a75d717894f7e8066c6e9ddbb15c7" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45543 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74bd37a79daf47b796b54e85fbf4e212" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40333 } health_report { overall_health: UNKNOWN } } }
I20260928 20:19:56.901513 11163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:56.899034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41416 (local address 127.3.18.2:45543)
0928 20:19:56.899127 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:56.899129 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:56.899138 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:19:56.899177 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:56.899178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:56.899196 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:56.899262 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:19:56.899360 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:56.899948 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:56.900722 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:56.900905 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:56.900958 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:56.901118 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:19:56.901221 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:19:56.901309 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:56.901405 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 20:19:56.901454 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 20:19:56.903416 11093 raft_consensus.cc:1261] T f5d74938fd7d47b3b4f3882b9257644e P f29fc7bde86e4c0faed6562bce8a47f4 [term 1 FOLLOWER]: Refusing update from remote peer 548a75d717894f7e8066c6e9ddbb15c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:19:56.903416 10813 raft_consensus.cc:1261] T f5d74938fd7d47b3b4f3882b9257644e P 74bd37a79daf47b796b54e85fbf4e212 [term 1 FOLLOWER]: Refusing update from remote peer 548a75d717894f7e8066c6e9ddbb15c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:19:56.903818 11172 consensus_queue.cc:1035] T f5d74938fd7d47b3b4f3882b9257644e P 548a75d717894f7e8066c6e9ddbb15c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74bd37a79daf47b796b54e85fbf4e212" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:19:56.903986 11177 consensus_queue.cc:1035] T f5d74938fd7d47b3b4f3882b9257644e P 548a75d717894f7e8066c6e9ddbb15c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:19:56.906984 11180 mvcc.cc:205] Tried to move back new op lower bound from 7334407360113094656 to 7334407360058515456. Current Snapshot: MvccSnapshot[applied={T|T < 7334407360113094656 or (T in {7334407360113094656})}]
I20260928 20:19:56.907120 11181 mvcc.cc:205] Tried to move back new op lower bound from 7334407360113094656 to 7334407360058515456. Current Snapshot: MvccSnapshot[applied={T|T < 7334407360113094656 or (T in {7334407360113094656})}]
I20260928 20:19:56.908236 11183 mvcc.cc:205] Tried to move back new op lower bound from 7334407360113094656 to 7334407360058515456. Current Snapshot: MvccSnapshot[applied={T|T < 7334407360113094656}]
I20260928 20:19:56.909101 10656 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:19:56.909193 10656 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:19:56.909960 10656 catalog_manager.cc:5782] T 00000000000000000000000000000000 P abfa814b87a34399b12eef4a1de11f23: Sending DeleteTablet for 3 replicas of tablet f5d74938fd7d47b3b4f3882b9257644e
I20260928 20:19:56.910274 11073 tablet_service.cc:1514] Processing DeleteTablet for tablet f5d74938fd7d47b3b4f3882b9257644e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:19:56 UTC) from {username='oryx', principal='oryx/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:44328
I20260928 20:19:56.910315 10933 tablet_service.cc:1514] Processing DeleteTablet for tablet f5d74938fd7d47b3b4f3882b9257644e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:19:56 UTC) from {username='oryx', principal='oryx/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:41414
I20260928 20:19:56.910509 10793 tablet_service.cc:1514] Processing DeleteTablet for tablet f5d74938fd7d47b3b4f3882b9257644e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:19:56 UTC) from {username='oryx', principal='oryx/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:44584
I20260928 20:19:56.910604 10656 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40080:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 20:19:56.910720 11191 tablet_replica.cc:330] T f5d74938fd7d47b3b4f3882b9257644e P 74bd37a79daf47b796b54e85fbf4e212: stopping tablet replica
W20260928 20:19:56.910784 10656 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 20:19:56.910811 11191 raft_consensus.cc:2229] T f5d74938fd7d47b3b4f3882b9257644e P 74bd37a79daf47b796b54e85fbf4e212 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:19:56.910903 11191 raft_consensus.cc:2258] T f5d74938fd7d47b3b4f3882b9257644e P 74bd37a79daf47b796b54e85fbf4e212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:19:56.911341 11189 tablet_replica.cc:330] T f5d74938fd7d47b3b4f3882b9257644e P f29fc7bde86e4c0faed6562bce8a47f4: stopping tablet replica
I20260928 20:19:56.911363 11191 ts_tablet_manager.cc:1905] T f5d74938fd7d47b3b4f3882b9257644e P 74bd37a79daf47b796b54e85fbf4e212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:19:56.911511 11189 raft_consensus.cc:2229] T f5d74938fd7d47b3b4f3882b9257644e P f29fc7bde86e4c0faed6562bce8a47f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:19:56.911674 11189 raft_consensus.cc:2258] T f5d74938fd7d47b3b4f3882b9257644e P f29fc7bde86e4c0faed6562bce8a47f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:19:56.912154 11189 ts_tablet_manager.cc:1905] T f5d74938fd7d47b3b4f3882b9257644e P f29fc7bde86e4c0faed6562bce8a47f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:19:56.912460 10793 tablet_service.cc:1467] Processing CreateTablet for tablet 26fd66a08081460180c5d354099f87db (DEFAULT_TABLE table=test-table [id=45ab7a0a5fc54090a126fbaa9480bd39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:19:56.912575 10793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26fd66a08081460180c5d354099f87db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:56.913196 10933 tablet_service.cc:1467] Processing CreateTablet for tablet 26fd66a08081460180c5d354099f87db (DEFAULT_TABLE table=test-table [id=45ab7a0a5fc54090a126fbaa9480bd39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:19:56.913350 10933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26fd66a08081460180c5d354099f87db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:56.913406 11191 ts_tablet_manager.cc:1918] T f5d74938fd7d47b3b4f3882b9257644e P 74bd37a79daf47b796b54e85fbf4e212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:19:56.913480 11191 log.cc:1196] T f5d74938fd7d47b3b4f3882b9257644e P 74bd37a79daf47b796b54e85fbf4e212: Deleting WAL directory at /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/ts-0/wal/wals/f5d74938fd7d47b3b4f3882b9257644e
I20260928 20:19:56.913743 11191 ts_tablet_manager.cc:1939] T f5d74938fd7d47b3b4f3882b9257644e P 74bd37a79daf47b796b54e85fbf4e212: Deleting consensus metadata
I20260928 20:19:56.914535 11189 ts_tablet_manager.cc:1918] T f5d74938fd7d47b3b4f3882b9257644e P f29fc7bde86e4c0faed6562bce8a47f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:19:56.914674 11189 log.cc:1196] T f5d74938fd7d47b3b4f3882b9257644e P f29fc7bde86e4c0faed6562bce8a47f4: Deleting WAL directory at /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/ts-2/wal/wals/f5d74938fd7d47b3b4f3882b9257644e
I20260928 20:19:56.914753 11168 tablet_bootstrap.cc:492] T 26fd66a08081460180c5d354099f87db P 548a75d717894f7e8066c6e9ddbb15c7: Bootstrap starting.
I20260928 20:19:56.914822 11073 tablet_service.cc:1467] Processing CreateTablet for tablet 26fd66a08081460180c5d354099f87db (DEFAULT_TABLE table=test-table [id=45ab7a0a5fc54090a126fbaa9480bd39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:19:56.914953 11073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26fd66a08081460180c5d354099f87db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:56.915151 11189 ts_tablet_manager.cc:1939] T f5d74938fd7d47b3b4f3882b9257644e P f29fc7bde86e4c0faed6562bce8a47f4: Deleting consensus metadata
I20260928 20:19:56.915210 11168 tablet_bootstrap.cc:654] T 26fd66a08081460180c5d354099f87db P 548a75d717894f7e8066c6e9ddbb15c7: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:56.915668 10641 catalog_manager.cc:4841] TS f29fc7bde86e4c0faed6562bce8a47f4 (127.3.18.3:36917): tablet f5d74938fd7d47b3b4f3882b9257644e (table test-table [id=7e0b8aa227544de69188354c9278aab8]) successfully deleted
I20260928 20:19:56.916208 11168 tablet_bootstrap.cc:492] T 26fd66a08081460180c5d354099f87db P 548a75d717894f7e8066c6e9ddbb15c7: No bootstrap required, opened a new log
I20260928 20:19:56.916255 11168 ts_tablet_manager.cc:1397] T 26fd66a08081460180c5d354099f87db P 548a75d717894f7e8066c6e9ddbb15c7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 20:19:56.916281 11167 tablet_bootstrap.cc:492] T 26fd66a08081460180c5d354099f87db P f29fc7bde86e4c0faed6562bce8a47f4: Bootstrap starting.
I20260928 20:19:56.916391 11168 raft_consensus.cc:348] T 26fd66a08081460180c5d354099f87db P 548a75d717894f7e8066c6e9ddbb15c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bd37a79daf47b796b54e85fbf4e212" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40333 } } peers { permanent_uuid: "548a75d717894f7e8066c6e9ddbb15c7" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45543 } } peers { permanent_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36917 } }
I20260928 20:19:56.916458 11168 raft_consensus.cc:374] T 26fd66a08081460180c5d354099f87db P 548a75d717894f7e8066c6e9ddbb15c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:56.916478 11168 raft_consensus.cc:729] T 26fd66a08081460180c5d354099f87db P 548a75d717894f7e8066c6e9ddbb15c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548a75d717894f7e8066c6e9ddbb15c7, State: Initialized, Role: FOLLOWER
I20260928 20:19:56.916523 11168 consensus_queue.cc:260] T 26fd66a08081460180c5d354099f87db P 548a75d717894f7e8066c6e9ddbb15c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bd37a79daf47b796b54e85fbf4e212" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40333 } } peers { permanent_uuid: "548a75d717894f7e8066c6e9ddbb15c7" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45543 } } peers { permanent_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36917 } }
I20260928 20:19:56.916607 11168 ts_tablet_manager.cc:1428] T 26fd66a08081460180c5d354099f87db P 548a75d717894f7e8066c6e9ddbb15c7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:56.916666 11167 tablet_bootstrap.cc:654] T 26fd66a08081460180c5d354099f87db P f29fc7bde86e4c0faed6562bce8a47f4: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:56.916967 11190 tablet_replica.cc:330] T f5d74938fd7d47b3b4f3882b9257644e P 548a75d717894f7e8066c6e9ddbb15c7: stopping tablet replica
I20260928 20:19:56.917066 11173 tablet_bootstrap.cc:492] T 26fd66a08081460180c5d354099f87db P 74bd37a79daf47b796b54e85fbf4e212: Bootstrap starting.
I20260928 20:19:56.917095 11190 raft_consensus.cc:2229] T f5d74938fd7d47b3b4f3882b9257644e P 548a75d717894f7e8066c6e9ddbb15c7 [term 1 LEADER]: Raft consensus shutting down.
I20260928 20:19:56.917201 11190 raft_consensus.cc:2258] T f5d74938fd7d47b3b4f3882b9257644e P 548a75d717894f7e8066c6e9ddbb15c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:19:56.917446 11173 tablet_bootstrap.cc:654] T 26fd66a08081460180c5d354099f87db P 74bd37a79daf47b796b54e85fbf4e212: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:56.917881 10642 catalog_manager.cc:4841] TS 74bd37a79daf47b796b54e85fbf4e212 (127.3.18.1:40333): tablet f5d74938fd7d47b3b4f3882b9257644e (table test-table [id=7e0b8aa227544de69188354c9278aab8]) successfully deleted
I20260928 20:19:56.918020 11173 tablet_bootstrap.cc:492] T 26fd66a08081460180c5d354099f87db P 74bd37a79daf47b796b54e85fbf4e212: No bootstrap required, opened a new log
I20260928 20:19:56.918073 11173 ts_tablet_manager.cc:1397] T 26fd66a08081460180c5d354099f87db P 74bd37a79daf47b796b54e85fbf4e212: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 20:19:56.918223 11173 raft_consensus.cc:348] T 26fd66a08081460180c5d354099f87db P 74bd37a79daf47b796b54e85fbf4e212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bd37a79daf47b796b54e85fbf4e212" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40333 } } peers { permanent_uuid: "548a75d717894f7e8066c6e9ddbb15c7" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45543 } } peers { permanent_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36917 } }
I20260928 20:19:56.918306 11173 raft_consensus.cc:374] T 26fd66a08081460180c5d354099f87db P 74bd37a79daf47b796b54e85fbf4e212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:56.918334 11173 raft_consensus.cc:729] T 26fd66a08081460180c5d354099f87db P 74bd37a79daf47b796b54e85fbf4e212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74bd37a79daf47b796b54e85fbf4e212, State: Initialized, Role: FOLLOWER
I20260928 20:19:56.918383 11173 consensus_queue.cc:260] T 26fd66a08081460180c5d354099f87db P 74bd37a79daf47b796b54e85fbf4e212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bd37a79daf47b796b54e85fbf4e212" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40333 } } peers { permanent_uuid: "548a75d717894f7e8066c6e9ddbb15c7" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45543 } } peers { permanent_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36917 } }
I20260928 20:19:56.918525 11190 ts_tablet_manager.cc:1905] T f5d74938fd7d47b3b4f3882b9257644e P 548a75d717894f7e8066c6e9ddbb15c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:19:56.918629 11167 tablet_bootstrap.cc:492] T 26fd66a08081460180c5d354099f87db P f29fc7bde86e4c0faed6562bce8a47f4: No bootstrap required, opened a new log
I20260928 20:19:56.918684 11167 ts_tablet_manager.cc:1397] T 26fd66a08081460180c5d354099f87db P f29fc7bde86e4c0faed6562bce8a47f4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 20:19:56.918723 11173 ts_tablet_manager.cc:1428] T 26fd66a08081460180c5d354099f87db P 74bd37a79daf47b796b54e85fbf4e212: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 20:19:56.918840 11167 raft_consensus.cc:348] T 26fd66a08081460180c5d354099f87db P f29fc7bde86e4c0faed6562bce8a47f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bd37a79daf47b796b54e85fbf4e212" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40333 } } peers { permanent_uuid: "548a75d717894f7e8066c6e9ddbb15c7" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45543 } } peers { permanent_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36917 } }
I20260928 20:19:56.918912 11167 raft_consensus.cc:374] T 26fd66a08081460180c5d354099f87db P f29fc7bde86e4c0faed6562bce8a47f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:56.918936 11167 raft_consensus.cc:729] T 26fd66a08081460180c5d354099f87db P f29fc7bde86e4c0faed6562bce8a47f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f29fc7bde86e4c0faed6562bce8a47f4, State: Initialized, Role: FOLLOWER
I20260928 20:19:56.918995 11167 consensus_queue.cc:260] T 26fd66a08081460180c5d354099f87db P f29fc7bde86e4c0faed6562bce8a47f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bd37a79daf47b796b54e85fbf4e212" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40333 } } peers { permanent_uuid: "548a75d717894f7e8066c6e9ddbb15c7" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45543 } } peers { permanent_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36917 } }
I20260928 20:19:56.919090 11167 ts_tablet_manager.cc:1428] T 26fd66a08081460180c5d354099f87db P f29fc7bde86e4c0faed6562bce8a47f4: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 20:19:56.920447 11190 ts_tablet_manager.cc:1918] T f5d74938fd7d47b3b4f3882b9257644e P 548a75d717894f7e8066c6e9ddbb15c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:19:56.920526 11190 log.cc:1196] T f5d74938fd7d47b3b4f3882b9257644e P 548a75d717894f7e8066c6e9ddbb15c7: Deleting WAL directory at /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/ts-1/wal/wals/f5d74938fd7d47b3b4f3882b9257644e
I20260928 20:19:56.920779 11190 ts_tablet_manager.cc:1939] T f5d74938fd7d47b3b4f3882b9257644e P 548a75d717894f7e8066c6e9ddbb15c7: Deleting consensus metadata
I20260928 20:19:56.921293 10642 catalog_manager.cc:4841] TS 548a75d717894f7e8066c6e9ddbb15c7 (127.3.18.2:45543): tablet f5d74938fd7d47b3b4f3882b9257644e (table test-table [id=7e0b8aa227544de69188354c9278aab8]) successfully deleted
I20260928 20:19:57.175062 11177 raft_consensus.cc:482] T 26fd66a08081460180c5d354099f87db P 548a75d717894f7e8066c6e9ddbb15c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:19:57.175184 11177 raft_consensus.cc:504] T 26fd66a08081460180c5d354099f87db P 548a75d717894f7e8066c6e9ddbb15c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bd37a79daf47b796b54e85fbf4e212" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40333 } } peers { permanent_uuid: "548a75d717894f7e8066c6e9ddbb15c7" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45543 } } peers { permanent_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36917 } }
I20260928 20:19:57.175381 11177 leader_election.cc:288] T 26fd66a08081460180c5d354099f87db P 548a75d717894f7e8066c6e9ddbb15c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74bd37a79daf47b796b54e85fbf4e212 (127.3.18.1:40333), f29fc7bde86e4c0faed6562bce8a47f4 (127.3.18.3:36917)
I20260928 20:19:57.175741 11093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26fd66a08081460180c5d354099f87db" candidate_uuid: "548a75d717894f7e8066c6e9ddbb15c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" is_pre_election: true
I20260928 20:19:57.175901 11093 raft_consensus.cc:2454] T 26fd66a08081460180c5d354099f87db P f29fc7bde86e4c0faed6562bce8a47f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548a75d717894f7e8066c6e9ddbb15c7 in term 0.
I20260928 20:19:57.175750 10813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26fd66a08081460180c5d354099f87db" candidate_uuid: "548a75d717894f7e8066c6e9ddbb15c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74bd37a79daf47b796b54e85fbf4e212" is_pre_election: true
I20260928 20:19:57.175992 10813 raft_consensus.cc:2454] T 26fd66a08081460180c5d354099f87db P 74bd37a79daf47b796b54e85fbf4e212 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548a75d717894f7e8066c6e9ddbb15c7 in term 0.
I20260928 20:19:57.176069 10885 leader_election.cc:302] T 26fd66a08081460180c5d354099f87db P 548a75d717894f7e8066c6e9ddbb15c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548a75d717894f7e8066c6e9ddbb15c7, f29fc7bde86e4c0faed6562bce8a47f4; no voters:
I20260928 20:19:57.176170 11177 raft_consensus.cc:2781] T 26fd66a08081460180c5d354099f87db P 548a75d717894f7e8066c6e9ddbb15c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 20:19:57.176198 11177 raft_consensus.cc:482] T 26fd66a08081460180c5d354099f87db P 548a75d717894f7e8066c6e9ddbb15c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 20:19:57.176224 11177 raft_consensus.cc:3037] T 26fd66a08081460180c5d354099f87db P 548a75d717894f7e8066c6e9ddbb15c7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:57.176944 11177 raft_consensus.cc:504] T 26fd66a08081460180c5d354099f87db P 548a75d717894f7e8066c6e9ddbb15c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bd37a79daf47b796b54e85fbf4e212" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40333 } } peers { permanent_uuid: "548a75d717894f7e8066c6e9ddbb15c7" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45543 } } peers { permanent_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36917 } }
I20260928 20:19:57.177104 11177 leader_election.cc:288] T 26fd66a08081460180c5d354099f87db P 548a75d717894f7e8066c6e9ddbb15c7 [CANDIDATE]: Term 1 election: Requested vote from peers 74bd37a79daf47b796b54e85fbf4e212 (127.3.18.1:40333), f29fc7bde86e4c0faed6562bce8a47f4 (127.3.18.3:36917)
I20260928 20:19:57.177312 10813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26fd66a08081460180c5d354099f87db" candidate_uuid: "548a75d717894f7e8066c6e9ddbb15c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74bd37a79daf47b796b54e85fbf4e212"
I20260928 20:19:57.177290 11093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26fd66a08081460180c5d354099f87db" candidate_uuid: "548a75d717894f7e8066c6e9ddbb15c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f29fc7bde86e4c0faed6562bce8a47f4"
I20260928 20:19:57.177385 10813 raft_consensus.cc:3037] T 26fd66a08081460180c5d354099f87db P 74bd37a79daf47b796b54e85fbf4e212 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:57.177385 11093 raft_consensus.cc:3037] T 26fd66a08081460180c5d354099f87db P f29fc7bde86e4c0faed6562bce8a47f4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:57.178180 11093 raft_consensus.cc:2454] T 26fd66a08081460180c5d354099f87db P f29fc7bde86e4c0faed6562bce8a47f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548a75d717894f7e8066c6e9ddbb15c7 in term 1.
I20260928 20:19:57.178180 10813 raft_consensus.cc:2454] T 26fd66a08081460180c5d354099f87db P 74bd37a79daf47b796b54e85fbf4e212 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548a75d717894f7e8066c6e9ddbb15c7 in term 1.
I20260928 20:19:57.178435 10885 leader_election.cc:302] T 26fd66a08081460180c5d354099f87db P 548a75d717894f7e8066c6e9ddbb15c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548a75d717894f7e8066c6e9ddbb15c7, f29fc7bde86e4c0faed6562bce8a47f4; no voters:
I20260928 20:19:57.178602 11177 raft_consensus.cc:2781] T 26fd66a08081460180c5d354099f87db P 548a75d717894f7e8066c6e9ddbb15c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:19:57.178668 11177 raft_consensus.cc:686] T 26fd66a08081460180c5d354099f87db P 548a75d717894f7e8066c6e9ddbb15c7 [term 1 LEADER]: Becoming Leader. State: Replica: 548a75d717894f7e8066c6e9ddbb15c7, State: Running, Role: LEADER
I20260928 20:19:57.178726 11177 consensus_queue.cc:237] T 26fd66a08081460180c5d354099f87db P 548a75d717894f7e8066c6e9ddbb15c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bd37a79daf47b796b54e85fbf4e212" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40333 } } peers { permanent_uuid: "548a75d717894f7e8066c6e9ddbb15c7" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45543 } } peers { permanent_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36917 } }
I20260928 20:19:57.179450 10655 catalog_manager.cc:5495] T 26fd66a08081460180c5d354099f87db P 548a75d717894f7e8066c6e9ddbb15c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 548a75d717894f7e8066c6e9ddbb15c7 (127.3.18.2). New cstate: current_term: 1 leader_uuid: "548a75d717894f7e8066c6e9ddbb15c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bd37a79daf47b796b54e85fbf4e212" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "548a75d717894f7e8066c6e9ddbb15c7" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45543 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36917 } health_report { overall_health: UNKNOWN } } }
Sep 28 20:19:57 dist-test-slave-b83v krb5kdc[10611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626796, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.18.62@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM
I20260928 20:19:57.242749 11202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:57.220835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40094 (local address 127.3.18.62:40593)
0928 20:19:57.221045 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:57.221047 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:57.221057 (+ 10us) server_negotiation.cc:407] Connection header received
0928 20:19:57.221083 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:57.221083 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:57.221094 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:57.221175 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:57.221713 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:57.222265 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:57.223160 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:57.223381 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:57.223453 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:57.239962 (+ 16509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:57.239964 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:57.239965 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:57.239972 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:57.241588 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:57.242023 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:57.242024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:57.242024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:57.242085 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:57.242275 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:57.242275 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:57.242276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:57.242403 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:57.242470 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:57.242594 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 20:19:57.242676 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":138,"threads_started":1}
I20260928 20:19:57.242836 11201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:57.220736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40593 (local address 127.0.0.1:40094)
0928 20:19:57.220907 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:57.220919 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:57.220980 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:57.221190 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:57.221191 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:57.221569 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:19:57.221620 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:57.221623 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:57.222387 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:57.222388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:57.223017 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:57.223020 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:57.223537 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:57.223539 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:57.223601 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:57.237631 (+ 14030us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:19:57.237638 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:19:57.239721 (+ 2083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:19:57.241688 (+ 1967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:57.241700 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:57.241704 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:57.241933 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:57.242157 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:19:57.242157 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:19:57.242158 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:19:57.242217 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:19:57.242466 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:19:57.242474 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:19:57.242522 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 20:19:57.242575 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 20:19:57.242749 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":70,"thread_start_us":109,"threads_started":1}
I20260928 20:19:57.245045 10655 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:40094:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 20:19:57.245287 10655 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 20:19:57.247093 10933 tablet_service.cc:1467] Processing CreateTablet for tablet 4d85e7e552494c118abab0e6f8828cad (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b572b6b959764305af195274e7687a30]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 20:19:57.247107 11073 tablet_service.cc:1467] Processing CreateTablet for tablet 4d85e7e552494c118abab0e6f8828cad (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b572b6b959764305af195274e7687a30]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 20:19:57.247097 10793 tablet_service.cc:1467] Processing CreateTablet for tablet 4d85e7e552494c118abab0e6f8828cad (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b572b6b959764305af195274e7687a30]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 20:19:57.247273 11073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d85e7e552494c118abab0e6f8828cad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:57.247274 10933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d85e7e552494c118abab0e6f8828cad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:57.247273 10793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d85e7e552494c118abab0e6f8828cad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:57.248854 11167 tablet_bootstrap.cc:492] T 4d85e7e552494c118abab0e6f8828cad P f29fc7bde86e4c0faed6562bce8a47f4: Bootstrap starting.
I20260928 20:19:57.248875 11173 tablet_bootstrap.cc:492] T 4d85e7e552494c118abab0e6f8828cad P 74bd37a79daf47b796b54e85fbf4e212: Bootstrap starting.
I20260928 20:19:57.249110 11168 tablet_bootstrap.cc:492] T 4d85e7e552494c118abab0e6f8828cad P 548a75d717894f7e8066c6e9ddbb15c7: Bootstrap starting.
I20260928 20:19:57.249326 11167 tablet_bootstrap.cc:654] T 4d85e7e552494c118abab0e6f8828cad P f29fc7bde86e4c0faed6562bce8a47f4: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:57.249364 11173 tablet_bootstrap.cc:654] T 4d85e7e552494c118abab0e6f8828cad P 74bd37a79daf47b796b54e85fbf4e212: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:57.249449 11168 tablet_bootstrap.cc:654] T 4d85e7e552494c118abab0e6f8828cad P 548a75d717894f7e8066c6e9ddbb15c7: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:57.250015 11167 tablet_bootstrap.cc:492] T 4d85e7e552494c118abab0e6f8828cad P f29fc7bde86e4c0faed6562bce8a47f4: No bootstrap required, opened a new log
I20260928 20:19:57.250058 11168 tablet_bootstrap.cc:492] T 4d85e7e552494c118abab0e6f8828cad P 548a75d717894f7e8066c6e9ddbb15c7: No bootstrap required, opened a new log
I20260928 20:19:57.250072 11167 ts_tablet_manager.cc:1397] T 4d85e7e552494c118abab0e6f8828cad P f29fc7bde86e4c0faed6562bce8a47f4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 20:19:57.250097 11168 ts_tablet_manager.cc:1397] T 4d85e7e552494c118abab0e6f8828cad P 548a75d717894f7e8066c6e9ddbb15c7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 20:19:57.250208 11168 raft_consensus.cc:348] T 4d85e7e552494c118abab0e6f8828cad P 548a75d717894f7e8066c6e9ddbb15c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bd37a79daf47b796b54e85fbf4e212" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40333 } } peers { permanent_uuid: "548a75d717894f7e8066c6e9ddbb15c7" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45543 } } peers { permanent_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36917 } }
I20260928 20:19:57.250211 11167 raft_consensus.cc:348] T 4d85e7e552494c118abab0e6f8828cad P f29fc7bde86e4c0faed6562bce8a47f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bd37a79daf47b796b54e85fbf4e212" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40333 } } peers { permanent_uuid: "548a75d717894f7e8066c6e9ddbb15c7" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45543 } } peers { permanent_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36917 } }
I20260928 20:19:57.250289 11168 raft_consensus.cc:374] T 4d85e7e552494c118abab0e6f8828cad P 548a75d717894f7e8066c6e9ddbb15c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:57.250289 11167 raft_consensus.cc:374] T 4d85e7e552494c118abab0e6f8828cad P f29fc7bde86e4c0faed6562bce8a47f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:57.250311 11167 raft_consensus.cc:729] T 4d85e7e552494c118abab0e6f8828cad P f29fc7bde86e4c0faed6562bce8a47f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f29fc7bde86e4c0faed6562bce8a47f4, State: Initialized, Role: FOLLOWER
I20260928 20:19:57.250311 11168 raft_consensus.cc:729] T 4d85e7e552494c118abab0e6f8828cad P 548a75d717894f7e8066c6e9ddbb15c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548a75d717894f7e8066c6e9ddbb15c7, State: Initialized, Role: FOLLOWER
I20260928 20:19:57.250360 11168 consensus_queue.cc:260] T 4d85e7e552494c118abab0e6f8828cad P 548a75d717894f7e8066c6e9ddbb15c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bd37a79daf47b796b54e85fbf4e212" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40333 } } peers { permanent_uuid: "548a75d717894f7e8066c6e9ddbb15c7" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45543 } } peers { permanent_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36917 } }
I20260928 20:19:57.250391 11167 consensus_queue.cc:260] T 4d85e7e552494c118abab0e6f8828cad P f29fc7bde86e4c0faed6562bce8a47f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bd37a79daf47b796b54e85fbf4e212" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40333 } } peers { permanent_uuid: "548a75d717894f7e8066c6e9ddbb15c7" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45543 } } peers { permanent_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36917 } }
I20260928 20:19:57.250465 11168 ts_tablet_manager.cc:1428] T 4d85e7e552494c118abab0e6f8828cad P 548a75d717894f7e8066c6e9ddbb15c7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:57.250492 11167 ts_tablet_manager.cc:1428] T 4d85e7e552494c118abab0e6f8828cad P f29fc7bde86e4c0faed6562bce8a47f4: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 20:19:57.250576 11177 tablet_replica.cc:439] T 4d85e7e552494c118abab0e6f8828cad P 548a75d717894f7e8066c6e9ddbb15c7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bd37a79daf47b796b54e85fbf4e212" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40333 } } peers { permanent_uuid: "548a75d717894f7e8066c6e9ddbb15c7" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45543 } } peers { permanent_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36917 } } }
I20260928 20:19:57.250604 11171 tablet_replica.cc:439] T 4d85e7e552494c118abab0e6f8828cad P f29fc7bde86e4c0faed6562bce8a47f4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bd37a79daf47b796b54e85fbf4e212" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40333 } } peers { permanent_uuid: "548a75d717894f7e8066c6e9ddbb15c7" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45543 } } peers { permanent_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36917 } } }
I20260928 20:19:57.250684 11177 tablet_replica.cc:442] T 4d85e7e552494c118abab0e6f8828cad P 548a75d717894f7e8066c6e9ddbb15c7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 20:19:57.250707 11171 tablet_replica.cc:442] T 4d85e7e552494c118abab0e6f8828cad P f29fc7bde86e4c0faed6562bce8a47f4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 20:19:57.251354 11173 tablet_bootstrap.cc:492] T 4d85e7e552494c118abab0e6f8828cad P 74bd37a79daf47b796b54e85fbf4e212: No bootstrap required, opened a new log
I20260928 20:19:57.251433 11173 ts_tablet_manager.cc:1397] T 4d85e7e552494c118abab0e6f8828cad P 74bd37a79daf47b796b54e85fbf4e212: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 20:19:57.251556 11173 raft_consensus.cc:348] T 4d85e7e552494c118abab0e6f8828cad P 74bd37a79daf47b796b54e85fbf4e212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bd37a79daf47b796b54e85fbf4e212" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40333 } } peers { permanent_uuid: "548a75d717894f7e8066c6e9ddbb15c7" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45543 } } peers { permanent_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36917 } }
I20260928 20:19:57.251612 11173 raft_consensus.cc:374] T 4d85e7e552494c118abab0e6f8828cad P 74bd37a79daf47b796b54e85fbf4e212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:57.251626 11173 raft_consensus.cc:729] T 4d85e7e552494c118abab0e6f8828cad P 74bd37a79daf47b796b54e85fbf4e212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74bd37a79daf47b796b54e85fbf4e212, State: Initialized, Role: FOLLOWER
I20260928 20:19:57.251683 11173 consensus_queue.cc:260] T 4d85e7e552494c118abab0e6f8828cad P 74bd37a79daf47b796b54e85fbf4e212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bd37a79daf47b796b54e85fbf4e212" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40333 } } peers { permanent_uuid: "548a75d717894f7e8066c6e9ddbb15c7" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45543 } } peers { permanent_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36917 } }
I20260928 20:19:57.251797 11173 ts_tablet_manager.cc:1428] T 4d85e7e552494c118abab0e6f8828cad P 74bd37a79daf47b796b54e85fbf4e212: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 20:19:57.251860 11175 tablet_replica.cc:439] T 4d85e7e552494c118abab0e6f8828cad P 74bd37a79daf47b796b54e85fbf4e212: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bd37a79daf47b796b54e85fbf4e212" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40333 } } peers { permanent_uuid: "548a75d717894f7e8066c6e9ddbb15c7" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45543 } } peers { permanent_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36917 } } }
I20260928 20:19:57.251999 11175 tablet_replica.cc:442] T 4d85e7e552494c118abab0e6f8828cad P 74bd37a79daf47b796b54e85fbf4e212: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 20:19:57.290096 11177 raft_consensus.cc:482] T 4d85e7e552494c118abab0e6f8828cad P 548a75d717894f7e8066c6e9ddbb15c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:19:57.290223 11177 raft_consensus.cc:504] T 4d85e7e552494c118abab0e6f8828cad P 548a75d717894f7e8066c6e9ddbb15c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bd37a79daf47b796b54e85fbf4e212" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40333 } } peers { permanent_uuid: "548a75d717894f7e8066c6e9ddbb15c7" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45543 } } peers { permanent_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36917 } }
I20260928 20:19:57.290428 11177 leader_election.cc:288] T 4d85e7e552494c118abab0e6f8828cad P 548a75d717894f7e8066c6e9ddbb15c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74bd37a79daf47b796b54e85fbf4e212 (127.3.18.1:40333), f29fc7bde86e4c0faed6562bce8a47f4 (127.3.18.3:36917)
I20260928 20:19:57.290663 10813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d85e7e552494c118abab0e6f8828cad" candidate_uuid: "548a75d717894f7e8066c6e9ddbb15c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74bd37a79daf47b796b54e85fbf4e212" is_pre_election: true
I20260928 20:19:57.290665 11093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d85e7e552494c118abab0e6f8828cad" candidate_uuid: "548a75d717894f7e8066c6e9ddbb15c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" is_pre_election: true
I20260928 20:19:57.290774 11093 raft_consensus.cc:2454] T 4d85e7e552494c118abab0e6f8828cad P f29fc7bde86e4c0faed6562bce8a47f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548a75d717894f7e8066c6e9ddbb15c7 in term 0.
I20260928 20:19:57.290791 10813 raft_consensus.cc:2454] T 4d85e7e552494c118abab0e6f8828cad P 74bd37a79daf47b796b54e85fbf4e212 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548a75d717894f7e8066c6e9ddbb15c7 in term 0.
I20260928 20:19:57.290949 10885 leader_election.cc:302] T 4d85e7e552494c118abab0e6f8828cad P 548a75d717894f7e8066c6e9ddbb15c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548a75d717894f7e8066c6e9ddbb15c7, f29fc7bde86e4c0faed6562bce8a47f4; no voters:
I20260928 20:19:57.291065 11177 raft_consensus.cc:2781] T 4d85e7e552494c118abab0e6f8828cad P 548a75d717894f7e8066c6e9ddbb15c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 20:19:57.291136 11177 raft_consensus.cc:482] T 4d85e7e552494c118abab0e6f8828cad P 548a75d717894f7e8066c6e9ddbb15c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 20:19:57.291167 11177 raft_consensus.cc:3037] T 4d85e7e552494c118abab0e6f8828cad P 548a75d717894f7e8066c6e9ddbb15c7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:57.292011 11177 raft_consensus.cc:504] T 4d85e7e552494c118abab0e6f8828cad P 548a75d717894f7e8066c6e9ddbb15c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bd37a79daf47b796b54e85fbf4e212" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40333 } } peers { permanent_uuid: "548a75d717894f7e8066c6e9ddbb15c7" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45543 } } peers { permanent_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36917 } }
I20260928 20:19:57.292193 11177 leader_election.cc:288] T 4d85e7e552494c118abab0e6f8828cad P 548a75d717894f7e8066c6e9ddbb15c7 [CANDIDATE]: Term 1 election: Requested vote from peers 74bd37a79daf47b796b54e85fbf4e212 (127.3.18.1:40333), f29fc7bde86e4c0faed6562bce8a47f4 (127.3.18.3:36917)
I20260928 20:19:57.292375 10813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d85e7e552494c118abab0e6f8828cad" candidate_uuid: "548a75d717894f7e8066c6e9ddbb15c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74bd37a79daf47b796b54e85fbf4e212"
I20260928 20:19:57.292456 10813 raft_consensus.cc:3037] T 4d85e7e552494c118abab0e6f8828cad P 74bd37a79daf47b796b54e85fbf4e212 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:57.292388 11093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d85e7e552494c118abab0e6f8828cad" candidate_uuid: "548a75d717894f7e8066c6e9ddbb15c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f29fc7bde86e4c0faed6562bce8a47f4"
I20260928 20:19:57.292488 11093 raft_consensus.cc:3037] T 4d85e7e552494c118abab0e6f8828cad P f29fc7bde86e4c0faed6562bce8a47f4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:57.293226 11093 raft_consensus.cc:2454] T 4d85e7e552494c118abab0e6f8828cad P f29fc7bde86e4c0faed6562bce8a47f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548a75d717894f7e8066c6e9ddbb15c7 in term 1.
I20260928 20:19:57.293244 10813 raft_consensus.cc:2454] T 4d85e7e552494c118abab0e6f8828cad P 74bd37a79daf47b796b54e85fbf4e212 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548a75d717894f7e8066c6e9ddbb15c7 in term 1.
I20260928 20:19:57.293457 10885 leader_election.cc:302] T 4d85e7e552494c118abab0e6f8828cad P 548a75d717894f7e8066c6e9ddbb15c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548a75d717894f7e8066c6e9ddbb15c7, f29fc7bde86e4c0faed6562bce8a47f4; no voters:
I20260928 20:19:57.293548 11177 raft_consensus.cc:2781] T 4d85e7e552494c118abab0e6f8828cad P 548a75d717894f7e8066c6e9ddbb15c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:19:57.293591 11177 raft_consensus.cc:686] T 4d85e7e552494c118abab0e6f8828cad P 548a75d717894f7e8066c6e9ddbb15c7 [term 1 LEADER]: Becoming Leader. State: Replica: 548a75d717894f7e8066c6e9ddbb15c7, State: Running, Role: LEADER
I20260928 20:19:57.293665 11177 consensus_queue.cc:237] T 4d85e7e552494c118abab0e6f8828cad P 548a75d717894f7e8066c6e9ddbb15c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bd37a79daf47b796b54e85fbf4e212" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40333 } } peers { permanent_uuid: "548a75d717894f7e8066c6e9ddbb15c7" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45543 } } peers { permanent_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36917 } }
I20260928 20:19:57.293978 11172 tablet_replica.cc:439] T 4d85e7e552494c118abab0e6f8828cad P 548a75d717894f7e8066c6e9ddbb15c7: TxnStatusTablet state changed. Reason: New leader 548a75d717894f7e8066c6e9ddbb15c7. Latest consensus state: current_term: 1 leader_uuid: "548a75d717894f7e8066c6e9ddbb15c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bd37a79daf47b796b54e85fbf4e212" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40333 } } peers { permanent_uuid: "548a75d717894f7e8066c6e9ddbb15c7" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45543 } } peers { permanent_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36917 } } }
I20260928 20:19:57.294066 11172 tablet_replica.cc:442] T 4d85e7e552494c118abab0e6f8828cad P 548a75d717894f7e8066c6e9ddbb15c7: This TxnStatusTablet replica's current role is: LEADER
I20260928 20:19:57.294236 11207 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 20:19:57.294353 11207 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 20:19:57.294337 10655 catalog_manager.cc:5495] T 4d85e7e552494c118abab0e6f8828cad P 548a75d717894f7e8066c6e9ddbb15c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 548a75d717894f7e8066c6e9ddbb15c7 (127.3.18.2). New cstate: current_term: 1 leader_uuid: "548a75d717894f7e8066c6e9ddbb15c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bd37a79daf47b796b54e85fbf4e212" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "548a75d717894f7e8066c6e9ddbb15c7" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45543 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36917 } health_report { overall_health: UNKNOWN } } }
I20260928 20:19:57.301501 11163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:57.298531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41418 (local address 127.3.18.2:45543)
0928 20:19:57.298589 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:57.298592 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:57.298607 (+ 15us) server_negotiation.cc:407] Connection header received
0928 20:19:57.298715 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:57.298716 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:57.298725 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:57.298785 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:19:57.298943 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:57.299601 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:57.300802 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:57.300991 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:57.301044 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:57.301271 (+ 227us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:19:57.301299 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:19:57.301338 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:57.301418 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 20:19:57.301439 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 20:19:57.301571 11201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:57.298459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.2:45543 (local address 127.0.0.1:41418)
0928 20:19:57.298569 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:57.298581 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:57.298627 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:57.298816 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:57.298817 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:57.298819 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 20:19:57.298875 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:57.298877 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:57.299692 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:57.299693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:57.300691 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:57.300696 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:57.301075 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:57.301076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:57.301153 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:57.301181 (+ 28us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 20:19:57.301355 (+ 174us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 20:19:57.301362 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 20:19:57.301416 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 20:19:57.301475 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 20:19:57.302735 11093 raft_consensus.cc:1261] T 4d85e7e552494c118abab0e6f8828cad P f29fc7bde86e4c0faed6562bce8a47f4 [term 1 FOLLOWER]: Refusing update from remote peer 548a75d717894f7e8066c6e9ddbb15c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:19:57.302801 10813 raft_consensus.cc:1261] T 4d85e7e552494c118abab0e6f8828cad P 74bd37a79daf47b796b54e85fbf4e212 [term 1 FOLLOWER]: Refusing update from remote peer 548a75d717894f7e8066c6e9ddbb15c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:19:57.302990 11177 consensus_queue.cc:1035] T 4d85e7e552494c118abab0e6f8828cad P 548a75d717894f7e8066c6e9ddbb15c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:19:57.303121 11172 consensus_queue.cc:1035] T 4d85e7e552494c118abab0e6f8828cad P 548a75d717894f7e8066c6e9ddbb15c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74bd37a79daf47b796b54e85fbf4e212" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:19:57.303664 11175 tablet_replica.cc:439] T 4d85e7e552494c118abab0e6f8828cad P 74bd37a79daf47b796b54e85fbf4e212: TxnStatusTablet state changed. Reason: New leader 548a75d717894f7e8066c6e9ddbb15c7. Latest consensus state: current_term: 1 leader_uuid: "548a75d717894f7e8066c6e9ddbb15c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bd37a79daf47b796b54e85fbf4e212" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40333 } } peers { permanent_uuid: "548a75d717894f7e8066c6e9ddbb15c7" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45543 } } peers { permanent_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36917 } } }
I20260928 20:19:57.303812 11175 tablet_replica.cc:442] T 4d85e7e552494c118abab0e6f8828cad P 74bd37a79daf47b796b54e85fbf4e212: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 20:19:57.303821 11171 tablet_replica.cc:439] T 4d85e7e552494c118abab0e6f8828cad P f29fc7bde86e4c0faed6562bce8a47f4: TxnStatusTablet state changed. Reason: New leader 548a75d717894f7e8066c6e9ddbb15c7. Latest consensus state: current_term: 1 leader_uuid: "548a75d717894f7e8066c6e9ddbb15c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bd37a79daf47b796b54e85fbf4e212" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40333 } } peers { permanent_uuid: "548a75d717894f7e8066c6e9ddbb15c7" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45543 } } peers { permanent_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36917 } } }
I20260928 20:19:57.303929 11171 tablet_replica.cc:442] T 4d85e7e552494c118abab0e6f8828cad P f29fc7bde86e4c0faed6562bce8a47f4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 20:19:57.304477 11177 tablet_replica.cc:439] T 4d85e7e552494c118abab0e6f8828cad P 548a75d717894f7e8066c6e9ddbb15c7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "548a75d717894f7e8066c6e9ddbb15c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bd37a79daf47b796b54e85fbf4e212" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40333 } } peers { permanent_uuid: "548a75d717894f7e8066c6e9ddbb15c7" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45543 } } peers { permanent_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36917 } } }
I20260928 20:19:57.304580 11177 tablet_replica.cc:442] T 4d85e7e552494c118abab0e6f8828cad P 548a75d717894f7e8066c6e9ddbb15c7: This TxnStatusTablet replica's current role is: LEADER
I20260928 20:19:57.305152 11172 tablet_replica.cc:439] T 4d85e7e552494c118abab0e6f8828cad P 548a75d717894f7e8066c6e9ddbb15c7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "548a75d717894f7e8066c6e9ddbb15c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bd37a79daf47b796b54e85fbf4e212" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40333 } } peers { permanent_uuid: "548a75d717894f7e8066c6e9ddbb15c7" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45543 } } peers { permanent_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36917 } } }
I20260928 20:19:57.305246 11172 tablet_replica.cc:442] T 4d85e7e552494c118abab0e6f8828cad P 548a75d717894f7e8066c6e9ddbb15c7: This TxnStatusTablet replica's current role is: LEADER
I20260928 20:19:57.305397 11175 tablet_replica.cc:439] T 4d85e7e552494c118abab0e6f8828cad P 74bd37a79daf47b796b54e85fbf4e212: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "548a75d717894f7e8066c6e9ddbb15c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bd37a79daf47b796b54e85fbf4e212" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40333 } } peers { permanent_uuid: "548a75d717894f7e8066c6e9ddbb15c7" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45543 } } peers { permanent_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36917 } } }
I20260928 20:19:57.305497 11175 tablet_replica.cc:442] T 4d85e7e552494c118abab0e6f8828cad P 74bd37a79daf47b796b54e85fbf4e212: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 20:19:57.305742 11171 tablet_replica.cc:439] T 4d85e7e552494c118abab0e6f8828cad P f29fc7bde86e4c0faed6562bce8a47f4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "548a75d717894f7e8066c6e9ddbb15c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bd37a79daf47b796b54e85fbf4e212" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40333 } } peers { permanent_uuid: "548a75d717894f7e8066c6e9ddbb15c7" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45543 } } peers { permanent_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36917 } } }
I20260928 20:19:57.305871 11171 tablet_replica.cc:442] T 4d85e7e552494c118abab0e6f8828cad P f29fc7bde86e4c0faed6562bce8a47f4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 20:19:57.313295 11163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:57.310277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:36545 (local address 127.3.18.2:45543)
0928 20:19:57.310319 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:57.310321 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:57.310525 (+ 204us) server_negotiation.cc:407] Connection header received
0928 20:19:57.310586 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:57.310587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:57.310598 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:57.310649 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:19:57.310783 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:57.311241 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:57.312398 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:57.312615 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:57.312685 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:57.312906 (+ 221us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:19:57.313009 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:19:57.313081 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:57.313185 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 20:19:57.313206 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 20:19:57.313313 11214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:57.310248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.2:45543 (local address 127.3.18.2:36545)
0928 20:19:57.310435 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:57.310449 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:57.310524 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:57.310667 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:57.310668 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:57.310670 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 20:19:57.310725 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:57.310727 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:57.311318 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:57.311319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:57.312298 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:57.312301 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:57.312705 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:57.312706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:57.312764 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:57.312786 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 20:19:57.313097 (+ 311us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 20:19:57.313099 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 20:19:57.313152 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 20:19:57.313201 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":95,"threads_started":1}
I20260928 20:19:57.315891 10813 raft_consensus.cc:1261] T 26fd66a08081460180c5d354099f87db P 74bd37a79daf47b796b54e85fbf4e212 [term 1 FOLLOWER]: Refusing update from remote peer 548a75d717894f7e8066c6e9ddbb15c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:19:57.315917 11093 raft_consensus.cc:1261] T 26fd66a08081460180c5d354099f87db P f29fc7bde86e4c0faed6562bce8a47f4 [term 1 FOLLOWER]: Refusing update from remote peer 548a75d717894f7e8066c6e9ddbb15c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:19:57.316136 11177 consensus_queue.cc:1035] T 26fd66a08081460180c5d354099f87db P 548a75d717894f7e8066c6e9ddbb15c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74bd37a79daf47b796b54e85fbf4e212" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:19:57.316255 11172 consensus_queue.cc:1035] T 26fd66a08081460180c5d354099f87db P 548a75d717894f7e8066c6e9ddbb15c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:19:57.327162 10655 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:19:57.327282 10655 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:19:57.327981 10655 catalog_manager.cc:5782] T 00000000000000000000000000000000 P abfa814b87a34399b12eef4a1de11f23: Sending DeleteTablet for 3 replicas of tablet 26fd66a08081460180c5d354099f87db
I20260928 20:19:57.328358 10932 tablet_service.cc:1514] Processing DeleteTablet for tablet 26fd66a08081460180c5d354099f87db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:19:57 UTC) from {username='oryx', principal='oryx/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:41414
I20260928 20:19:57.328362 10793 tablet_service.cc:1514] Processing DeleteTablet for tablet 26fd66a08081460180c5d354099f87db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:19:57 UTC) from {username='oryx', principal='oryx/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:44584
I20260928 20:19:57.328363 11073 tablet_service.cc:1514] Processing DeleteTablet for tablet 26fd66a08081460180c5d354099f87db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:19:57 UTC) from {username='oryx', principal='oryx/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:44328
I20260928 20:19:57.328456 11191 tablet_replica.cc:330] T 26fd66a08081460180c5d354099f87db P 74bd37a79daf47b796b54e85fbf4e212: stopping tablet replica
I20260928 20:19:57.328465 11190 tablet_replica.cc:330] T 26fd66a08081460180c5d354099f87db P 548a75d717894f7e8066c6e9ddbb15c7: stopping tablet replica
I20260928 20:19:57.328503 11191 raft_consensus.cc:2229] T 26fd66a08081460180c5d354099f87db P 74bd37a79daf47b796b54e85fbf4e212 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:19:57.328540 11191 raft_consensus.cc:2258] T 26fd66a08081460180c5d354099f87db P 74bd37a79daf47b796b54e85fbf4e212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:19:57.328531 11190 raft_consensus.cc:2229] T 26fd66a08081460180c5d354099f87db P 548a75d717894f7e8066c6e9ddbb15c7 [term 1 LEADER]: Raft consensus shutting down.
I20260928 20:19:57.328581 11190 raft_consensus.cc:2258] T 26fd66a08081460180c5d354099f87db P 548a75d717894f7e8066c6e9ddbb15c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:19:57.328641 11189 tablet_replica.cc:330] T 26fd66a08081460180c5d354099f87db P f29fc7bde86e4c0faed6562bce8a47f4: stopping tablet replica
I20260928 20:19:57.328686 11189 raft_consensus.cc:2229] T 26fd66a08081460180c5d354099f87db P f29fc7bde86e4c0faed6562bce8a47f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:19:57.328725 11189 raft_consensus.cc:2258] T 26fd66a08081460180c5d354099f87db P f29fc7bde86e4c0faed6562bce8a47f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:19:57.329069 11189 ts_tablet_manager.cc:1905] T 26fd66a08081460180c5d354099f87db P f29fc7bde86e4c0faed6562bce8a47f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:19:57.330848 11190 ts_tablet_manager.cc:1905] T 26fd66a08081460180c5d354099f87db P 548a75d717894f7e8066c6e9ddbb15c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:19:57.330945 11189 ts_tablet_manager.cc:1918] T 26fd66a08081460180c5d354099f87db P f29fc7bde86e4c0faed6562bce8a47f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 20:19:57.331035 11189 log.cc:1196] T 26fd66a08081460180c5d354099f87db P f29fc7bde86e4c0faed6562bce8a47f4: Deleting WAL directory at /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/ts-2/wal/wals/26fd66a08081460180c5d354099f87db
I20260928 20:19:57.331245 11189 ts_tablet_manager.cc:1939] T 26fd66a08081460180c5d354099f87db P f29fc7bde86e4c0faed6562bce8a47f4: Deleting consensus metadata
I20260928 20:19:57.331700 10641 catalog_manager.cc:4841] TS f29fc7bde86e4c0faed6562bce8a47f4 (127.3.18.3:36917): tablet 26fd66a08081460180c5d354099f87db (table test-table [id=45ab7a0a5fc54090a126fbaa9480bd39]) successfully deleted
I20260928 20:19:57.332371 11190 ts_tablet_manager.cc:1918] T 26fd66a08081460180c5d354099f87db P 548a75d717894f7e8066c6e9ddbb15c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 20:19:57.332437 11190 log.cc:1196] T 26fd66a08081460180c5d354099f87db P 548a75d717894f7e8066c6e9ddbb15c7: Deleting WAL directory at /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/ts-1/wal/wals/26fd66a08081460180c5d354099f87db
I20260928 20:19:57.332662 11190 ts_tablet_manager.cc:1939] T 26fd66a08081460180c5d354099f87db P 548a75d717894f7e8066c6e9ddbb15c7: Deleting consensus metadata
I20260928 20:19:57.333217 10642 catalog_manager.cc:4841] TS 548a75d717894f7e8066c6e9ddbb15c7 (127.3.18.2:45543): tablet 26fd66a08081460180c5d354099f87db (table test-table [id=45ab7a0a5fc54090a126fbaa9480bd39]) successfully deleted
I20260928 20:19:57.333614 11191 ts_tablet_manager.cc:1905] T 26fd66a08081460180c5d354099f87db P 74bd37a79daf47b796b54e85fbf4e212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:19:57.335166 11191 ts_tablet_manager.cc:1918] T 26fd66a08081460180c5d354099f87db P 74bd37a79daf47b796b54e85fbf4e212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 20:19:57.335222 11191 log.cc:1196] T 26fd66a08081460180c5d354099f87db P 74bd37a79daf47b796b54e85fbf4e212: Deleting WAL directory at /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/ts-0/wal/wals/26fd66a08081460180c5d354099f87db
I20260928 20:19:57.335459 11191 ts_tablet_manager.cc:1939] T 26fd66a08081460180c5d354099f87db P 74bd37a79daf47b796b54e85fbf4e212: Deleting consensus metadata
I20260928 20:19:57.335925 10642 catalog_manager.cc:4841] TS 74bd37a79daf47b796b54e85fbf4e212 (127.3.18.1:40333): tablet 26fd66a08081460180c5d354099f87db (table test-table [id=45ab7a0a5fc54090a126fbaa9480bd39]) successfully deleted
Sep 28 20:19:57 dist-test-slave-b83v krb5kdc[10611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626797, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 20:19:57 dist-test-slave-b83v krb5kdc[10611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:19:57 dist-test-slave-b83v krb5kdc[10611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 20:19:57.359449 11202 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:19:57.353516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40110 (local address 127.3.18.62:40593)
0928 20:19:57.353595 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:57.353598 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:57.353721 (+ 123us) server_negotiation.cc:407] Connection header received
0928 20:19:57.353804 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:57.353805 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:57.353816 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:57.353900 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:57.354482 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:57.354997 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:57.355839 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:57.356181 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:57.356270 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:57.359377 (+ 3107us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40110: BlockingRecv error: recv got EOF from 127.0.0.1:40110 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 28 20:19:57 dist-test-slave-b83v krb5kdc[10611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626797, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM
I20260928 20:19:57.368826 11202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:57.360485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40112 (local address 127.3.18.62:40593)
0928 20:19:57.360552 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:57.360555 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:57.360648 (+ 93us) server_negotiation.cc:407] Connection header received
0928 20:19:57.360713 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:57.360714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:57.360723 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:57.360810 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:19:57.361389 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:57.361986 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:57.363024 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:57.363333 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:57.363389 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:57.366004 (+ 2615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:19:57.366006 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:19:57.366006 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:19:57.366014 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:19:57.367509 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:57.368023 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:57.368025 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:57.368025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:57.368089 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:19:57.368292 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:19:57.368294 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:19:57.368294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:19:57.368416 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:19:57.368533 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:57.368634 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 20:19:57.368720 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 20:19:57.370391 10655 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40112:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 20:19:57.370572 10655 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 20:19:57.372062 10793 tablet_service.cc:1467] Processing CreateTablet for tablet 3fc2966413c24172884a6e52359be181 (DEFAULT_TABLE table=test-table [id=59f75f9b39224beb89f97de72efb33f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:19:57.372092 10932 tablet_service.cc:1467] Processing CreateTablet for tablet 3fc2966413c24172884a6e52359be181 (DEFAULT_TABLE table=test-table [id=59f75f9b39224beb89f97de72efb33f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:19:57.372212 10793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fc2966413c24172884a6e52359be181. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:57.372249 10932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fc2966413c24172884a6e52359be181. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:57.373387 11168 tablet_bootstrap.cc:492] T 3fc2966413c24172884a6e52359be181 P 548a75d717894f7e8066c6e9ddbb15c7: Bootstrap starting.
I20260928 20:19:57.373585 11073 tablet_service.cc:1467] Processing CreateTablet for tablet 3fc2966413c24172884a6e52359be181 (DEFAULT_TABLE table=test-table [id=59f75f9b39224beb89f97de72efb33f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:19:57.373718 11073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fc2966413c24172884a6e52359be181. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:57.373754 11168 tablet_bootstrap.cc:654] T 3fc2966413c24172884a6e52359be181 P 548a75d717894f7e8066c6e9ddbb15c7: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:57.374366 11168 tablet_bootstrap.cc:492] T 3fc2966413c24172884a6e52359be181 P 548a75d717894f7e8066c6e9ddbb15c7: No bootstrap required, opened a new log
I20260928 20:19:57.374436 11168 ts_tablet_manager.cc:1397] T 3fc2966413c24172884a6e52359be181 P 548a75d717894f7e8066c6e9ddbb15c7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 20:19:57.374570 11168 raft_consensus.cc:348] T 3fc2966413c24172884a6e52359be181 P 548a75d717894f7e8066c6e9ddbb15c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bd37a79daf47b796b54e85fbf4e212" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40333 } } peers { permanent_uuid: "548a75d717894f7e8066c6e9ddbb15c7" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45543 } } peers { permanent_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36917 } }
I20260928 20:19:57.374663 11168 raft_consensus.cc:374] T 3fc2966413c24172884a6e52359be181 P 548a75d717894f7e8066c6e9ddbb15c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:57.374692 11168 raft_consensus.cc:729] T 3fc2966413c24172884a6e52359be181 P 548a75d717894f7e8066c6e9ddbb15c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548a75d717894f7e8066c6e9ddbb15c7, State: Initialized, Role: FOLLOWER
I20260928 20:19:57.374732 11168 consensus_queue.cc:260] T 3fc2966413c24172884a6e52359be181 P 548a75d717894f7e8066c6e9ddbb15c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bd37a79daf47b796b54e85fbf4e212" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40333 } } peers { permanent_uuid: "548a75d717894f7e8066c6e9ddbb15c7" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45543 } } peers { permanent_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36917 } }
I20260928 20:19:57.374825 11168 ts_tablet_manager.cc:1428] T 3fc2966413c24172884a6e52359be181 P 548a75d717894f7e8066c6e9ddbb15c7: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 20:19:57.374967 11173 tablet_bootstrap.cc:492] T 3fc2966413c24172884a6e52359be181 P 74bd37a79daf47b796b54e85fbf4e212: Bootstrap starting.
I20260928 20:19:57.375387 11173 tablet_bootstrap.cc:654] T 3fc2966413c24172884a6e52359be181 P 74bd37a79daf47b796b54e85fbf4e212: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:57.375747 11167 tablet_bootstrap.cc:492] T 3fc2966413c24172884a6e52359be181 P f29fc7bde86e4c0faed6562bce8a47f4: Bootstrap starting.
I20260928 20:19:57.376205 11167 tablet_bootstrap.cc:654] T 3fc2966413c24172884a6e52359be181 P f29fc7bde86e4c0faed6562bce8a47f4: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:57.376689 11173 tablet_bootstrap.cc:492] T 3fc2966413c24172884a6e52359be181 P 74bd37a79daf47b796b54e85fbf4e212: No bootstrap required, opened a new log
I20260928 20:19:57.376749 11173 ts_tablet_manager.cc:1397] T 3fc2966413c24172884a6e52359be181 P 74bd37a79daf47b796b54e85fbf4e212: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 20:19:57.376899 11173 raft_consensus.cc:348] T 3fc2966413c24172884a6e52359be181 P 74bd37a79daf47b796b54e85fbf4e212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bd37a79daf47b796b54e85fbf4e212" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40333 } } peers { permanent_uuid: "548a75d717894f7e8066c6e9ddbb15c7" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45543 } } peers { permanent_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36917 } }
I20260928 20:19:57.376991 11173 raft_consensus.cc:374] T 3fc2966413c24172884a6e52359be181 P 74bd37a79daf47b796b54e85fbf4e212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:57.377014 11173 raft_consensus.cc:729] T 3fc2966413c24172884a6e52359be181 P 74bd37a79daf47b796b54e85fbf4e212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74bd37a79daf47b796b54e85fbf4e212, State: Initialized, Role: FOLLOWER
I20260928 20:19:57.377059 11173 consensus_queue.cc:260] T 3fc2966413c24172884a6e52359be181 P 74bd37a79daf47b796b54e85fbf4e212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bd37a79daf47b796b54e85fbf4e212" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40333 } } peers { permanent_uuid: "548a75d717894f7e8066c6e9ddbb15c7" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45543 } } peers { permanent_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36917 } }
I20260928 20:19:57.377164 11173 ts_tablet_manager.cc:1428] T 3fc2966413c24172884a6e52359be181 P 74bd37a79daf47b796b54e85fbf4e212: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:57.377382 11167 tablet_bootstrap.cc:492] T 3fc2966413c24172884a6e52359be181 P f29fc7bde86e4c0faed6562bce8a47f4: No bootstrap required, opened a new log
I20260928 20:19:57.377435 11167 ts_tablet_manager.cc:1397] T 3fc2966413c24172884a6e52359be181 P f29fc7bde86e4c0faed6562bce8a47f4: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 20:19:57.377609 11167 raft_consensus.cc:348] T 3fc2966413c24172884a6e52359be181 P f29fc7bde86e4c0faed6562bce8a47f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bd37a79daf47b796b54e85fbf4e212" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40333 } } peers { permanent_uuid: "548a75d717894f7e8066c6e9ddbb15c7" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45543 } } peers { permanent_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36917 } }
I20260928 20:19:57.377684 11167 raft_consensus.cc:374] T 3fc2966413c24172884a6e52359be181 P f29fc7bde86e4c0faed6562bce8a47f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:57.377707 11167 raft_consensus.cc:729] T 3fc2966413c24172884a6e52359be181 P f29fc7bde86e4c0faed6562bce8a47f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f29fc7bde86e4c0faed6562bce8a47f4, State: Initialized, Role: FOLLOWER
I20260928 20:19:57.377748 11167 consensus_queue.cc:260] T 3fc2966413c24172884a6e52359be181 P f29fc7bde86e4c0faed6562bce8a47f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bd37a79daf47b796b54e85fbf4e212" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40333 } } peers { permanent_uuid: "548a75d717894f7e8066c6e9ddbb15c7" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45543 } } peers { permanent_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36917 } }
I20260928 20:19:57.377831 11167 ts_tablet_manager.cc:1428] T 3fc2966413c24172884a6e52359be181 P f29fc7bde86e4c0faed6562bce8a47f4: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 20:19:57.406219 11219 raft_consensus.cc:482] T 3fc2966413c24172884a6e52359be181 P 548a75d717894f7e8066c6e9ddbb15c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:19:57.406343 11219 raft_consensus.cc:504] T 3fc2966413c24172884a6e52359be181 P 548a75d717894f7e8066c6e9ddbb15c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bd37a79daf47b796b54e85fbf4e212" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40333 } } peers { permanent_uuid: "548a75d717894f7e8066c6e9ddbb15c7" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45543 } } peers { permanent_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36917 } }
I20260928 20:19:57.406519 11219 leader_election.cc:288] T 3fc2966413c24172884a6e52359be181 P 548a75d717894f7e8066c6e9ddbb15c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74bd37a79daf47b796b54e85fbf4e212 (127.3.18.1:40333), f29fc7bde86e4c0faed6562bce8a47f4 (127.3.18.3:36917)
I20260928 20:19:57.406831 10813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fc2966413c24172884a6e52359be181" candidate_uuid: "548a75d717894f7e8066c6e9ddbb15c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74bd37a79daf47b796b54e85fbf4e212" is_pre_election: true
I20260928 20:19:57.406931 11093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fc2966413c24172884a6e52359be181" candidate_uuid: "548a75d717894f7e8066c6e9ddbb15c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" is_pre_election: true
I20260928 20:19:57.406980 10813 raft_consensus.cc:2454] T 3fc2966413c24172884a6e52359be181 P 74bd37a79daf47b796b54e85fbf4e212 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548a75d717894f7e8066c6e9ddbb15c7 in term 0.
I20260928 20:19:57.407024 11093 raft_consensus.cc:2454] T 3fc2966413c24172884a6e52359be181 P f29fc7bde86e4c0faed6562bce8a47f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548a75d717894f7e8066c6e9ddbb15c7 in term 0.
I20260928 20:19:57.407197 10886 leader_election.cc:302] T 3fc2966413c24172884a6e52359be181 P 548a75d717894f7e8066c6e9ddbb15c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548a75d717894f7e8066c6e9ddbb15c7, 74bd37a79daf47b796b54e85fbf4e212; no voters:
I20260928 20:19:57.407333 11219 raft_consensus.cc:2781] T 3fc2966413c24172884a6e52359be181 P 548a75d717894f7e8066c6e9ddbb15c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 20:19:57.407379 11219 raft_consensus.cc:482] T 3fc2966413c24172884a6e52359be181 P 548a75d717894f7e8066c6e9ddbb15c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 20:19:57.407405 11219 raft_consensus.cc:3037] T 3fc2966413c24172884a6e52359be181 P 548a75d717894f7e8066c6e9ddbb15c7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:57.408262 11219 raft_consensus.cc:504] T 3fc2966413c24172884a6e52359be181 P 548a75d717894f7e8066c6e9ddbb15c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bd37a79daf47b796b54e85fbf4e212" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40333 } } peers { permanent_uuid: "548a75d717894f7e8066c6e9ddbb15c7" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45543 } } peers { permanent_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36917 } }
I20260928 20:19:57.408449 11219 leader_election.cc:288] T 3fc2966413c24172884a6e52359be181 P 548a75d717894f7e8066c6e9ddbb15c7 [CANDIDATE]: Term 1 election: Requested vote from peers 74bd37a79daf47b796b54e85fbf4e212 (127.3.18.1:40333), f29fc7bde86e4c0faed6562bce8a47f4 (127.3.18.3:36917)
I20260928 20:19:57.408694 11093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fc2966413c24172884a6e52359be181" candidate_uuid: "548a75d717894f7e8066c6e9ddbb15c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f29fc7bde86e4c0faed6562bce8a47f4"
I20260928 20:19:57.408694 10813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fc2966413c24172884a6e52359be181" candidate_uuid: "548a75d717894f7e8066c6e9ddbb15c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74bd37a79daf47b796b54e85fbf4e212"
I20260928 20:19:57.408792 11093 raft_consensus.cc:3037] T 3fc2966413c24172884a6e52359be181 P f29fc7bde86e4c0faed6562bce8a47f4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:57.408810 10813 raft_consensus.cc:3037] T 3fc2966413c24172884a6e52359be181 P 74bd37a79daf47b796b54e85fbf4e212 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:57.409528 10813 raft_consensus.cc:2454] T 3fc2966413c24172884a6e52359be181 P 74bd37a79daf47b796b54e85fbf4e212 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548a75d717894f7e8066c6e9ddbb15c7 in term 1.
I20260928 20:19:57.409564 11093 raft_consensus.cc:2454] T 3fc2966413c24172884a6e52359be181 P f29fc7bde86e4c0faed6562bce8a47f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548a75d717894f7e8066c6e9ddbb15c7 in term 1.
I20260928 20:19:57.409741 10885 leader_election.cc:302] T 3fc2966413c24172884a6e52359be181 P 548a75d717894f7e8066c6e9ddbb15c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548a75d717894f7e8066c6e9ddbb15c7, f29fc7bde86e4c0faed6562bce8a47f4; no voters:
I20260928 20:19:57.409827 11219 raft_consensus.cc:2781] T 3fc2966413c24172884a6e52359be181 P 548a75d717894f7e8066c6e9ddbb15c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:19:57.409878 11219 raft_consensus.cc:686] T 3fc2966413c24172884a6e52359be181 P 548a75d717894f7e8066c6e9ddbb15c7 [term 1 LEADER]: Becoming Leader. State: Replica: 548a75d717894f7e8066c6e9ddbb15c7, State: Running, Role: LEADER
I20260928 20:19:57.409940 11219 consensus_queue.cc:237] T 3fc2966413c24172884a6e52359be181 P 548a75d717894f7e8066c6e9ddbb15c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bd37a79daf47b796b54e85fbf4e212" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40333 } } peers { permanent_uuid: "548a75d717894f7e8066c6e9ddbb15c7" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45543 } } peers { permanent_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36917 } }
I20260928 20:19:57.410612 10656 catalog_manager.cc:5495] T 3fc2966413c24172884a6e52359be181 P 548a75d717894f7e8066c6e9ddbb15c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 548a75d717894f7e8066c6e9ddbb15c7 (127.3.18.2). New cstate: current_term: 1 leader_uuid: "548a75d717894f7e8066c6e9ddbb15c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bd37a79daf47b796b54e85fbf4e212" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "548a75d717894f7e8066c6e9ddbb15c7" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45543 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36917 } health_report { overall_health: UNKNOWN } } }
I20260928 20:19:57.425063 11163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:57.422333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41430 (local address 127.3.18.2:45543)
0928 20:19:57.422388 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:57.422390 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:57.422399 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:19:57.422464 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:57.422465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:57.422475 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:57.422524 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:19:57.422813 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:57.423406 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:57.424306 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:57.424514 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:57.424618 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:57.424772 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:19:57.424814 (+ 42us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:19:57.424875 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:57.424937 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 20:19:57.424964 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 20:19:57.426052 10813 raft_consensus.cc:1261] T 3fc2966413c24172884a6e52359be181 P 74bd37a79daf47b796b54e85fbf4e212 [term 1 FOLLOWER]: Refusing update from remote peer 548a75d717894f7e8066c6e9ddbb15c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:19:57.426273 11093 raft_consensus.cc:1261] T 3fc2966413c24172884a6e52359be181 P f29fc7bde86e4c0faed6562bce8a47f4 [term 1 FOLLOWER]: Refusing update from remote peer 548a75d717894f7e8066c6e9ddbb15c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:19:57.426368 11217 consensus_queue.cc:1035] T 3fc2966413c24172884a6e52359be181 P 548a75d717894f7e8066c6e9ddbb15c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74bd37a79daf47b796b54e85fbf4e212" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:19:57.426638 11217 consensus_queue.cc:1035] T 3fc2966413c24172884a6e52359be181 P 548a75d717894f7e8066c6e9ddbb15c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:19:57.429288 10655 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:19:57.429394 10655 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:19:57.430192 10655 catalog_manager.cc:5782] T 00000000000000000000000000000000 P abfa814b87a34399b12eef4a1de11f23: Sending DeleteTablet for 3 replicas of tablet 3fc2966413c24172884a6e52359be181
I20260928 20:19:57.430399 11073 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fc2966413c24172884a6e52359be181 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:19:57 UTC) from {username='oryx', principal='oryx/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:44328
I20260928 20:19:57.430490 11189 tablet_replica.cc:330] T 3fc2966413c24172884a6e52359be181 P f29fc7bde86e4c0faed6562bce8a47f4: stopping tablet replica
I20260928 20:19:57.430493 11072 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fc2966413c24172884a6e52359be181 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:19:57 UTC) from {username='oryx', principal='oryx/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:44328
I20260928 20:19:57.430634 11189 raft_consensus.cc:2229] T 3fc2966413c24172884a6e52359be181 P f29fc7bde86e4c0faed6562bce8a47f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:19:57.430699 11189 raft_consensus.cc:2258] T 3fc2966413c24172884a6e52359be181 P f29fc7bde86e4c0faed6562bce8a47f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:19:57.430691 10793 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fc2966413c24172884a6e52359be181 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:19:57 UTC) from {username='oryx', principal='oryx/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:44584
I20260928 20:19:57.430794 11191 tablet_replica.cc:330] T 3fc2966413c24172884a6e52359be181 P 74bd37a79daf47b796b54e85fbf4e212: stopping tablet replica
I20260928 20:19:57.430836 11191 raft_consensus.cc:2229] T 3fc2966413c24172884a6e52359be181 P 74bd37a79daf47b796b54e85fbf4e212 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:19:57.430881 11191 raft_consensus.cc:2258] T 3fc2966413c24172884a6e52359be181 P 74bd37a79daf47b796b54e85fbf4e212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:19:57.431039 11189 ts_tablet_manager.cc:1905] T 3fc2966413c24172884a6e52359be181 P f29fc7bde86e4c0faed6562bce8a47f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:19:57.431228 10932 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fc2966413c24172884a6e52359be181 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:19:57 UTC) from {username='oryx', principal='oryx/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:41414
I20260928 20:19:57.431339 11190 tablet_replica.cc:330] T 3fc2966413c24172884a6e52359be181 P 548a75d717894f7e8066c6e9ddbb15c7: stopping tablet replica
I20260928 20:19:57.431397 11190 raft_consensus.cc:2229] T 3fc2966413c24172884a6e52359be181 P 548a75d717894f7e8066c6e9ddbb15c7 [term 1 LEADER]: Raft consensus shutting down.
I20260928 20:19:57.431452 11190 raft_consensus.cc:2258] T 3fc2966413c24172884a6e52359be181 P 548a75d717894f7e8066c6e9ddbb15c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:19:57.431679 10656 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40112:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 20:19:57.431874 10656 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 20:19:57.432291 11191 ts_tablet_manager.cc:1905] T 3fc2966413c24172884a6e52359be181 P 74bd37a79daf47b796b54e85fbf4e212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:19:57.432837 11189 ts_tablet_manager.cc:1918] T 3fc2966413c24172884a6e52359be181 P f29fc7bde86e4c0faed6562bce8a47f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:19:57.432897 11189 log.cc:1196] T 3fc2966413c24172884a6e52359be181 P f29fc7bde86e4c0faed6562bce8a47f4: Deleting WAL directory at /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/ts-2/wal/wals/3fc2966413c24172884a6e52359be181
I20260928 20:19:57.433133 11189 ts_tablet_manager.cc:1939] T 3fc2966413c24172884a6e52359be181 P f29fc7bde86e4c0faed6562bce8a47f4: Deleting consensus metadata
I20260928 20:19:57.433372 11072 tablet_service.cc:1467] Processing CreateTablet for tablet 315325902da349d7ad2c076008b95975 (DEFAULT_TABLE table=test-table [id=a3b6f93cb9ee4e3dba4baa29d9a71d0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:19:57.433477 11072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 315325902da349d7ad2c076008b95975. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:57.433488 11190 ts_tablet_manager.cc:1905] T 3fc2966413c24172884a6e52359be181 P 548a75d717894f7e8066c6e9ddbb15c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:19:57.433844 10793 tablet_service.cc:1467] Processing CreateTablet for tablet 315325902da349d7ad2c076008b95975 (DEFAULT_TABLE table=test-table [id=a3b6f93cb9ee4e3dba4baa29d9a71d0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:19:57.434051 10932 tablet_service.cc:1467] Processing CreateTablet for tablet 315325902da349d7ad2c076008b95975 (DEFAULT_TABLE table=test-table [id=a3b6f93cb9ee4e3dba4baa29d9a71d0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:19:57.434149 10932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 315325902da349d7ad2c076008b95975. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:57.435106 11191 ts_tablet_manager.cc:1918] T 3fc2966413c24172884a6e52359be181 P 74bd37a79daf47b796b54e85fbf4e212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:19:57.435122 11190 ts_tablet_manager.cc:1918] T 3fc2966413c24172884a6e52359be181 P 548a75d717894f7e8066c6e9ddbb15c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:19:57.435169 11190 log.cc:1196] T 3fc2966413c24172884a6e52359be181 P 548a75d717894f7e8066c6e9ddbb15c7: Deleting WAL directory at /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/ts-1/wal/wals/3fc2966413c24172884a6e52359be181
I20260928 20:19:57.435169 11191 log.cc:1196] T 3fc2966413c24172884a6e52359be181 P 74bd37a79daf47b796b54e85fbf4e212: Deleting WAL directory at /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/ts-0/wal/wals/3fc2966413c24172884a6e52359be181
I20260928 20:19:57.435388 11191 ts_tablet_manager.cc:1939] T 3fc2966413c24172884a6e52359be181 P 74bd37a79daf47b796b54e85fbf4e212: Deleting consensus metadata
I20260928 20:19:57.435388 11190 ts_tablet_manager.cc:1939] T 3fc2966413c24172884a6e52359be181 P 548a75d717894f7e8066c6e9ddbb15c7: Deleting consensus metadata
I20260928 20:19:57.435434 11167 tablet_bootstrap.cc:492] T 315325902da349d7ad2c076008b95975 P f29fc7bde86e4c0faed6562bce8a47f4: Bootstrap starting.
I20260928 20:19:57.435928 10641 catalog_manager.cc:4841] TS f29fc7bde86e4c0faed6562bce8a47f4 (127.3.18.3:36917): tablet 3fc2966413c24172884a6e52359be181 (table test-table [id=59f75f9b39224beb89f97de72efb33f3]) successfully deleted
I20260928 20:19:57.436081 11167 tablet_bootstrap.cc:654] T 315325902da349d7ad2c076008b95975 P f29fc7bde86e4c0faed6562bce8a47f4: Neither blocks nor log segments found. Creating new log.
W20260928 20:19:57.436084 10641 catalog_manager.cc:4808] TS f29fc7bde86e4c0faed6562bce8a47f4 (127.3.18.3:36917): delete failed for tablet 3fc2966413c24172884a6e52359be181 because the tablet was not found. No further retry: Not found: Tablet not found: 3fc2966413c24172884a6e52359be181
I20260928 20:19:57.436409 11168 tablet_bootstrap.cc:492] T 315325902da349d7ad2c076008b95975 P 548a75d717894f7e8066c6e9ddbb15c7: Bootstrap starting.
I20260928 20:19:57.436710 11167 tablet_bootstrap.cc:492] T 315325902da349d7ad2c076008b95975 P f29fc7bde86e4c0faed6562bce8a47f4: No bootstrap required, opened a new log
I20260928 20:19:57.436754 11167 ts_tablet_manager.cc:1397] T 315325902da349d7ad2c076008b95975 P f29fc7bde86e4c0faed6562bce8a47f4: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 20:19:57.436795 11168 tablet_bootstrap.cc:654] T 315325902da349d7ad2c076008b95975 P 548a75d717894f7e8066c6e9ddbb15c7: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:57.436885 11167 raft_consensus.cc:348] T 315325902da349d7ad2c076008b95975 P f29fc7bde86e4c0faed6562bce8a47f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bd37a79daf47b796b54e85fbf4e212" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40333 } } peers { permanent_uuid: "548a75d717894f7e8066c6e9ddbb15c7" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45543 } } peers { permanent_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36917 } }
I20260928 20:19:57.436939 11167 raft_consensus.cc:374] T 315325902da349d7ad2c076008b95975 P f29fc7bde86e4c0faed6562bce8a47f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:57.436952 11167 raft_consensus.cc:729] T 315325902da349d7ad2c076008b95975 P f29fc7bde86e4c0faed6562bce8a47f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f29fc7bde86e4c0faed6562bce8a47f4, State: Initialized, Role: FOLLOWER
I20260928 20:19:57.437024 11167 consensus_queue.cc:260] T 315325902da349d7ad2c076008b95975 P f29fc7bde86e4c0faed6562bce8a47f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bd37a79daf47b796b54e85fbf4e212" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40333 } } peers { permanent_uuid: "548a75d717894f7e8066c6e9ddbb15c7" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45543 } } peers { permanent_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36917 } }
I20260928 20:19:57.437101 10793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 315325902da349d7ad2c076008b95975. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:19:57.437122 11167 ts_tablet_manager.cc:1428] T 315325902da349d7ad2c076008b95975 P f29fc7bde86e4c0faed6562bce8a47f4: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 20:19:57.437398 10642 catalog_manager.cc:4841] TS 74bd37a79daf47b796b54e85fbf4e212 (127.3.18.1:40333): tablet 3fc2966413c24172884a6e52359be181 (table test-table [id=59f75f9b39224beb89f97de72efb33f3]) successfully deleted
I20260928 20:19:57.438337 11173 tablet_bootstrap.cc:492] T 315325902da349d7ad2c076008b95975 P 74bd37a79daf47b796b54e85fbf4e212: Bootstrap starting.
I20260928 20:19:57.438776 10642 catalog_manager.cc:4841] TS 548a75d717894f7e8066c6e9ddbb15c7 (127.3.18.2:45543): tablet 3fc2966413c24172884a6e52359be181 (table test-table [id=59f75f9b39224beb89f97de72efb33f3]) successfully deleted
I20260928 20:19:57.438832 11173 tablet_bootstrap.cc:654] T 315325902da349d7ad2c076008b95975 P 74bd37a79daf47b796b54e85fbf4e212: Neither blocks nor log segments found. Creating new log.
I20260928 20:19:57.439152 11168 tablet_bootstrap.cc:492] T 315325902da349d7ad2c076008b95975 P 548a75d717894f7e8066c6e9ddbb15c7: No bootstrap required, opened a new log
I20260928 20:19:57.439216 11168 ts_tablet_manager.cc:1397] T 315325902da349d7ad2c076008b95975 P 548a75d717894f7e8066c6e9ddbb15c7: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 20:19:57.439374 11168 raft_consensus.cc:348] T 315325902da349d7ad2c076008b95975 P 548a75d717894f7e8066c6e9ddbb15c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bd37a79daf47b796b54e85fbf4e212" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40333 } } peers { permanent_uuid: "548a75d717894f7e8066c6e9ddbb15c7" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45543 } } peers { permanent_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36917 } }
I20260928 20:19:57.439464 11168 raft_consensus.cc:374] T 315325902da349d7ad2c076008b95975 P 548a75d717894f7e8066c6e9ddbb15c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:57.439505 11168 raft_consensus.cc:729] T 315325902da349d7ad2c076008b95975 P 548a75d717894f7e8066c6e9ddbb15c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548a75d717894f7e8066c6e9ddbb15c7, State: Initialized, Role: FOLLOWER
I20260928 20:19:57.439553 11168 consensus_queue.cc:260] T 315325902da349d7ad2c076008b95975 P 548a75d717894f7e8066c6e9ddbb15c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bd37a79daf47b796b54e85fbf4e212" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40333 } } peers { permanent_uuid: "548a75d717894f7e8066c6e9ddbb15c7" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45543 } } peers { permanent_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36917 } }
I20260928 20:19:57.439669 11168 ts_tablet_manager.cc:1428] T 315325902da349d7ad2c076008b95975 P 548a75d717894f7e8066c6e9ddbb15c7: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 20:19:57.439822 11173 tablet_bootstrap.cc:492] T 315325902da349d7ad2c076008b95975 P 74bd37a79daf47b796b54e85fbf4e212: No bootstrap required, opened a new log
I20260928 20:19:57.439869 11173 ts_tablet_manager.cc:1397] T 315325902da349d7ad2c076008b95975 P 74bd37a79daf47b796b54e85fbf4e212: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 20:19:57.440016 11173 raft_consensus.cc:348] T 315325902da349d7ad2c076008b95975 P 74bd37a79daf47b796b54e85fbf4e212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bd37a79daf47b796b54e85fbf4e212" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40333 } } peers { permanent_uuid: "548a75d717894f7e8066c6e9ddbb15c7" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45543 } } peers { permanent_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36917 } }
I20260928 20:19:57.440083 11173 raft_consensus.cc:374] T 315325902da349d7ad2c076008b95975 P 74bd37a79daf47b796b54e85fbf4e212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:19:57.440109 11173 raft_consensus.cc:729] T 315325902da349d7ad2c076008b95975 P 74bd37a79daf47b796b54e85fbf4e212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74bd37a79daf47b796b54e85fbf4e212, State: Initialized, Role: FOLLOWER
I20260928 20:19:57.440150 11173 consensus_queue.cc:260] T 315325902da349d7ad2c076008b95975 P 74bd37a79daf47b796b54e85fbf4e212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bd37a79daf47b796b54e85fbf4e212" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40333 } } peers { permanent_uuid: "548a75d717894f7e8066c6e9ddbb15c7" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45543 } } peers { permanent_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36917 } }
I20260928 20:19:57.440228 11173 ts_tablet_manager.cc:1428] T 315325902da349d7ad2c076008b95975 P 74bd37a79daf47b796b54e85fbf4e212: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 20:19:57.716259 11171 raft_consensus.cc:482] T 315325902da349d7ad2c076008b95975 P f29fc7bde86e4c0faed6562bce8a47f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:19:57.716364 11171 raft_consensus.cc:504] T 315325902da349d7ad2c076008b95975 P f29fc7bde86e4c0faed6562bce8a47f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bd37a79daf47b796b54e85fbf4e212" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40333 } } peers { permanent_uuid: "548a75d717894f7e8066c6e9ddbb15c7" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45543 } } peers { permanent_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36917 } }
I20260928 20:19:57.716686 11171 leader_election.cc:288] T 315325902da349d7ad2c076008b95975 P f29fc7bde86e4c0faed6562bce8a47f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74bd37a79daf47b796b54e85fbf4e212 (127.3.18.1:40333), 548a75d717894f7e8066c6e9ddbb15c7 (127.3.18.2:45543)
I20260928 20:19:57.719971 11163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:57.717241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:60407 (local address 127.3.18.2:45543)
0928 20:19:57.717284 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:57.717286 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:57.717299 (+ 13us) server_negotiation.cc:407] Connection header received
0928 20:19:57.717355 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:57.717356 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:57.717369 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:57.717427 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:19:57.717614 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:57.718096 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:57.719195 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:57.719597 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:57.719681 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:57.719695 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:57.719858 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 20:19:57.719899 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 20:19:57.720119 11243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:57.717013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:51525 (local address 127.3.18.1:40333)
0928 20:19:57.717139 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:57.717142 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:57.717156 (+ 14us) server_negotiation.cc:407] Connection header received
0928 20:19:57.717257 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:57.717257 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:57.717269 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:57.717338 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:19:57.717563 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:57.718108 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:57.719250 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:57.719700 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:57.719836 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:57.719849 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:57.719999 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 20:19:57.720043 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":69,"threads_started":1}
I20260928 20:19:57.720000 11244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:57.717133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.2:45543 (local address 127.3.18.3:60407)
0928 20:19:57.717227 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:57.717238 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:57.717293 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:57.717446 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:57.717447 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:57.717448 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:19:57.717533 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:57.717535 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:57.718217 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:57.718218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:57.719090 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:57.719093 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:57.719696 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:57.719697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:57.719777 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:57.719782 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 20:19:57.719851 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 20:19:57.719893 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81,"mutex_wait_us":24,"thread_start_us":52,"threads_started":1}
I20260928 20:19:57.720130 11242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:57.716907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.1:40333 (local address 127.3.18.3:51525)
0928 20:19:57.717062 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:57.717072 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:57.717174 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:57.717353 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:57.717354 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:57.717356 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:19:57.717418 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:57.717420 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:57.718251 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:57.718252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:57.719158 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:57.719161 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:57.719840 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:57.719841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:57.719902 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:57.719906 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:19:57.719991 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 20:19:57.720032 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":71,"thread_start_us":101,"threads_started":1}
I20260928 20:19:57.720636 10813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "315325902da349d7ad2c076008b95975" candidate_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74bd37a79daf47b796b54e85fbf4e212" is_pre_election: true
I20260928 20:19:57.720638 10953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "315325902da349d7ad2c076008b95975" candidate_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548a75d717894f7e8066c6e9ddbb15c7" is_pre_election: true
I20260928 20:19:57.720741 10813 raft_consensus.cc:2454] T 315325902da349d7ad2c076008b95975 P 74bd37a79daf47b796b54e85fbf4e212 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f29fc7bde86e4c0faed6562bce8a47f4 in term 0.
I20260928 20:19:57.720757 10953 raft_consensus.cc:2454] T 315325902da349d7ad2c076008b95975 P 548a75d717894f7e8066c6e9ddbb15c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f29fc7bde86e4c0faed6562bce8a47f4 in term 0.
I20260928 20:19:57.720911 11026 leader_election.cc:302] T 315325902da349d7ad2c076008b95975 P f29fc7bde86e4c0faed6562bce8a47f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74bd37a79daf47b796b54e85fbf4e212, f29fc7bde86e4c0faed6562bce8a47f4; no voters:
I20260928 20:19:57.721062 11171 raft_consensus.cc:2781] T 315325902da349d7ad2c076008b95975 P f29fc7bde86e4c0faed6562bce8a47f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 20:19:57.721103 11171 raft_consensus.cc:482] T 315325902da349d7ad2c076008b95975 P f29fc7bde86e4c0faed6562bce8a47f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 20:19:57.721125 11171 raft_consensus.cc:3037] T 315325902da349d7ad2c076008b95975 P f29fc7bde86e4c0faed6562bce8a47f4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:57.721794 11171 raft_consensus.cc:504] T 315325902da349d7ad2c076008b95975 P f29fc7bde86e4c0faed6562bce8a47f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bd37a79daf47b796b54e85fbf4e212" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40333 } } peers { permanent_uuid: "548a75d717894f7e8066c6e9ddbb15c7" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45543 } } peers { permanent_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36917 } }
I20260928 20:19:57.721926 11171 leader_election.cc:288] T 315325902da349d7ad2c076008b95975 P f29fc7bde86e4c0faed6562bce8a47f4 [CANDIDATE]: Term 1 election: Requested vote from peers 74bd37a79daf47b796b54e85fbf4e212 (127.3.18.1:40333), 548a75d717894f7e8066c6e9ddbb15c7 (127.3.18.2:45543)
I20260928 20:19:57.722121 10813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "315325902da349d7ad2c076008b95975" candidate_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74bd37a79daf47b796b54e85fbf4e212"
I20260928 20:19:57.722193 10813 raft_consensus.cc:3037] T 315325902da349d7ad2c076008b95975 P 74bd37a79daf47b796b54e85fbf4e212 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:57.722781 10813 raft_consensus.cc:2454] T 315325902da349d7ad2c076008b95975 P 74bd37a79daf47b796b54e85fbf4e212 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f29fc7bde86e4c0faed6562bce8a47f4 in term 1.
I20260928 20:19:57.722908 10953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "315325902da349d7ad2c076008b95975" candidate_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548a75d717894f7e8066c6e9ddbb15c7"
I20260928 20:19:57.722985 10953 raft_consensus.cc:3037] T 315325902da349d7ad2c076008b95975 P 548a75d717894f7e8066c6e9ddbb15c7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:19:57.722961 11026 leader_election.cc:302] T 315325902da349d7ad2c076008b95975 P f29fc7bde86e4c0faed6562bce8a47f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74bd37a79daf47b796b54e85fbf4e212, f29fc7bde86e4c0faed6562bce8a47f4; no voters:
I20260928 20:19:57.723138 11171 raft_consensus.cc:2781] T 315325902da349d7ad2c076008b95975 P f29fc7bde86e4c0faed6562bce8a47f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:19:57.723263 11171 raft_consensus.cc:686] T 315325902da349d7ad2c076008b95975 P f29fc7bde86e4c0faed6562bce8a47f4 [term 1 LEADER]: Becoming Leader. State: Replica: f29fc7bde86e4c0faed6562bce8a47f4, State: Running, Role: LEADER
I20260928 20:19:57.723335 11171 consensus_queue.cc:237] T 315325902da349d7ad2c076008b95975 P f29fc7bde86e4c0faed6562bce8a47f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bd37a79daf47b796b54e85fbf4e212" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40333 } } peers { permanent_uuid: "548a75d717894f7e8066c6e9ddbb15c7" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45543 } } peers { permanent_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36917 } }
I20260928 20:19:57.723932 10953 raft_consensus.cc:2454] T 315325902da349d7ad2c076008b95975 P 548a75d717894f7e8066c6e9ddbb15c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f29fc7bde86e4c0faed6562bce8a47f4 in term 1.
I20260928 20:19:57.724123 10655 catalog_manager.cc:5495] T 315325902da349d7ad2c076008b95975 P f29fc7bde86e4c0faed6562bce8a47f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to f29fc7bde86e4c0faed6562bce8a47f4 (127.3.18.3). New cstate: current_term: 1 leader_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bd37a79daf47b796b54e85fbf4e212" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "548a75d717894f7e8066c6e9ddbb15c7" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f29fc7bde86e4c0faed6562bce8a47f4" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 36917 } health_report { overall_health: HEALTHY } } }
I20260928 20:19:57.801079 11247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:57.798441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44332 (local address 127.3.18.3:36917)
0928 20:19:57.798550 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:57.798552 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:57.798560 (+ 8us) server_negotiation.cc:407] Connection header received
0928 20:19:57.798612 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:57.798613 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:57.798633 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:57.798708 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:19:57.798813 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:57.799407 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:57.800188 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:57.800453 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:57.800511 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:57.800680 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:19:57.800798 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:19:57.800858 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:57.800968 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 20:19:57.800987 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":65,"threads_started":1}
I20260928 20:19:57.806692 11163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:57.804014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:51773 (local address 127.3.18.2:45543)
0928 20:19:57.804076 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:57.804079 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:57.804224 (+ 145us) server_negotiation.cc:407] Connection header received
0928 20:19:57.804295 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:57.804297 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:57.804308 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:57.804380 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:19:57.804570 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:57.805062 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:57.805944 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:57.806142 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:57.806233 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:57.806412 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:19:57.806456 (+ 44us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:19:57.806517 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:57.806586 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 20:19:57.806609 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 20:19:57.806730 11250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:57.803884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.2:45543 (local address 127.3.18.3:51773)
0928 20:19:57.804157 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:57.804167 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:57.804241 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:57.804380 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:57.804381 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:57.804383 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 20:19:57.804462 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:57.804463 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:57.805162 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:57.805163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:57.805858 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:57.805861 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:57.806247 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:57.806248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:57.806313 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:57.806328 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 20:19:57.806519 (+ 191us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 20:19:57.806521 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 20:19:57.806581 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 20:19:57.806643 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":97,"threads_started":1}
I20260928 20:19:57.809405 10813 raft_consensus.cc:1261] T 315325902da349d7ad2c076008b95975 P 74bd37a79daf47b796b54e85fbf4e212 [term 1 FOLLOWER]: Refusing update from remote peer f29fc7bde86e4c0faed6562bce8a47f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:19:57.809502 10953 raft_consensus.cc:1261] T 315325902da349d7ad2c076008b95975 P 548a75d717894f7e8066c6e9ddbb15c7 [term 1 FOLLOWER]: Refusing update from remote peer f29fc7bde86e4c0faed6562bce8a47f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:19:57.809769 11171 consensus_queue.cc:1035] T 315325902da349d7ad2c076008b95975 P f29fc7bde86e4c0faed6562bce8a47f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74bd37a79daf47b796b54e85fbf4e212" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 40333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:19:57.809942 11245 consensus_queue.cc:1035] T 315325902da349d7ad2c076008b95975 P f29fc7bde86e4c0faed6562bce8a47f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548a75d717894f7e8066c6e9ddbb15c7" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:19:57.817610 11255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:57.815001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.3:36917 (local address 127.3.18.2:48213)
0928 20:19:57.815166 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:57.815178 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:19:57.815252 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:19:57.815415 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:19:57.815416 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:19:57.815418 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 20:19:57.815478 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:57.815480 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:57.816211 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:57.816212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:57.816791 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:19:57.816794 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:19:57.817247 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:19:57.817247 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:19:57.817312 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:57.817317 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 20:19:57.817460 (+ 143us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 20:19:57.817461 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 20:19:57.817505 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 20:19:57.817544 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":78,"threads_started":1}
I20260928 20:19:57.817637 11247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:57.815076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:48213 (local address 127.3.18.3:36917)
0928 20:19:57.815123 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 20:19:57.815124 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 20:19:57.815228 (+ 104us) server_negotiation.cc:407] Connection header received
0928 20:19:57.815330 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:19:57.815331 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:19:57.815344 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:19:57.815410 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:19:57.815557 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:57.816145 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:57.816876 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:19:57.817170 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:19:57.817238 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:19:57.817378 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:19:57.817414 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:19:57.817461 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 20:19:57.817565 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 20:19:57.817584 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 20:19:57.824141 10656 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:19:57.824260 10656 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:19:57.824901 10656 catalog_manager.cc:5782] T 00000000000000000000000000000000 P abfa814b87a34399b12eef4a1de11f23: Sending DeleteTablet for 3 replicas of tablet 315325902da349d7ad2c076008b95975
I20260928 20:19:57.825217 10793 tablet_service.cc:1514] Processing DeleteTablet for tablet 315325902da349d7ad2c076008b95975 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:19:57 UTC) from {username='oryx', principal='oryx/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:44584
I20260928 20:19:57.825318 11191 tablet_replica.cc:330] T 315325902da349d7ad2c076008b95975 P 74bd37a79daf47b796b54e85fbf4e212: stopping tablet replica
I20260928 20:19:57.825364 11191 raft_consensus.cc:2229] T 315325902da349d7ad2c076008b95975 P 74bd37a79daf47b796b54e85fbf4e212 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:19:57.825398 11191 raft_consensus.cc:2258] T 315325902da349d7ad2c076008b95975 P 74bd37a79daf47b796b54e85fbf4e212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:19:57.825392 11072 tablet_service.cc:1514] Processing DeleteTablet for tablet 315325902da349d7ad2c076008b95975 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:19:57 UTC) from {username='oryx', principal='oryx/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:44328
I20260928 20:19:57.825518 11189 tablet_replica.cc:330] T 315325902da349d7ad2c076008b95975 P f29fc7bde86e4c0faed6562bce8a47f4: stopping tablet replica
I20260928 20:19:57.825518 10932 tablet_service.cc:1514] Processing DeleteTablet for tablet 315325902da349d7ad2c076008b95975 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:19:57 UTC) from {username='oryx', principal='oryx/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:41414
I20260928 20:19:57.825572 11189 raft_consensus.cc:2229] T 315325902da349d7ad2c076008b95975 P f29fc7bde86e4c0faed6562bce8a47f4 [term 1 LEADER]: Raft consensus shutting down.
I20260928 20:19:57.825609 11190 tablet_replica.cc:330] T 315325902da349d7ad2c076008b95975 P 548a75d717894f7e8066c6e9ddbb15c7: stopping tablet replica
I20260928 20:19:57.825649 11190 raft_consensus.cc:2229] T 315325902da349d7ad2c076008b95975 P 548a75d717894f7e8066c6e9ddbb15c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:19:57.825657 11189 raft_consensus.cc:2258] T 315325902da349d7ad2c076008b95975 P f29fc7bde86e4c0faed6562bce8a47f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:19:57.825680 11190 raft_consensus.cc:2258] T 315325902da349d7ad2c076008b95975 P 548a75d717894f7e8066c6e9ddbb15c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:19:57.825883 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 10728
I20260928 20:19:57.826119 11189 ts_tablet_manager.cc:1905] T 315325902da349d7ad2c076008b95975 P f29fc7bde86e4c0faed6562bce8a47f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:19:57.828727 11189 ts_tablet_manager.cc:1918] T 315325902da349d7ad2c076008b95975 P f29fc7bde86e4c0faed6562bce8a47f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 20:19:57.828804 11189 log.cc:1196] T 315325902da349d7ad2c076008b95975 P f29fc7bde86e4c0faed6562bce8a47f4: Deleting WAL directory at /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/ts-2/wal/wals/315325902da349d7ad2c076008b95975
I20260928 20:19:57.829027 11189 ts_tablet_manager.cc:1939] T 315325902da349d7ad2c076008b95975 P f29fc7bde86e4c0faed6562bce8a47f4: Deleting consensus metadata
I20260928 20:19:57.829389 10641 catalog_manager.cc:4841] TS f29fc7bde86e4c0faed6562bce8a47f4 (127.3.18.3:36917): tablet 315325902da349d7ad2c076008b95975 (table test-table [id=a3b6f93cb9ee4e3dba4baa29d9a71d0e]) successfully deleted
I20260928 20:19:57.829532 11190 ts_tablet_manager.cc:1905] T 315325902da349d7ad2c076008b95975 P 548a75d717894f7e8066c6e9ddbb15c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:19:57.831050 11190 ts_tablet_manager.cc:1918] T 315325902da349d7ad2c076008b95975 P 548a75d717894f7e8066c6e9ddbb15c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 20:19:57.831125 11190 log.cc:1196] T 315325902da349d7ad2c076008b95975 P 548a75d717894f7e8066c6e9ddbb15c7: Deleting WAL directory at /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790626764250411-3144-0/minicluster-data/ts-1/wal/wals/315325902da349d7ad2c076008b95975
I20260928 20:19:57.831362 11190 ts_tablet_manager.cc:1939] T 315325902da349d7ad2c076008b95975 P 548a75d717894f7e8066c6e9ddbb15c7: Deleting consensus metadata
I20260928 20:19:57.831714 10642 catalog_manager.cc:4841] TS 548a75d717894f7e8066c6e9ddbb15c7 (127.3.18.2:45543): tablet 315325902da349d7ad2c076008b95975 (table test-table [id=a3b6f93cb9ee4e3dba4baa29d9a71d0e]) successfully deleted
W20260928 20:19:57.833247 10642 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.18.1:40333 (error 108)
I20260928 20:19:57.833431 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 10868
I20260928 20:19:57.834693 11258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:19:57.834470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.1:40333 (local address 127.0.0.1:44592)
0928 20:19:57.834577 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 20:19:57.834646 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.1:40333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":64,"threads_started":1}
W20260928 20:19:57.834928 10642 catalog_manager.cc:4584] TS 74bd37a79daf47b796b54e85fbf4e212 (127.3.18.1:40333): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 315325902da349d7ad2c076008b95975: Network error: Client connection negotiation failed: client connection to 127.3.18.1:40333: connect: Connection refused (error 111)
I20260928 20:19:57.842495 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 11008
I20260928 20:19:57.848096 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 10624
2026-09-28T20:19:57Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3118 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790626764250411-3144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:19:57 dist-test-slave-b83v krb5kdc[11261](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:19:57 dist-test-slave-b83v krb5kdc[11261](info): set up 2 sockets
Sep 28 20:19:57 dist-test-slave-b83v krb5kdc[11261](info): commencing operation
krb5kdc: starting...
W20260928 20:19:58.917097 3144 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Sep 28 20:19:59 dist-test-slave-b83v krb5kdc[11261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:19:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:19:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.18.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab.
Entry for principal kudu/127.3.18.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab.
I20260928 20:19:59.048143 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790626764250411-3144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790626764250411-3144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790626764250411-3144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:34661
--webserver_interface=127.3.18.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:38865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.18.62:34661
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:19:59.133085 11274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:19:59.133365 11274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:19:59.133394 11274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:19:59.134697 11274 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:19:59.134748 11274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:19:59.134763 11274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:19:59.134776 11274 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:19:59.134789 11274 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:19:59.136507 11274 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:38865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790626764250411-3144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790626764250411-3144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.18.62:34661
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:34661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.18.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790626764250411-3144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:19:59.136746 11274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:19:59.136935 11274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:19:59.143107 11279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:19:59.143131 11282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:59.143289 11274 server_base.cc:1034] running on GCE node
W20260928 20:19:59.143100 11280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:19:59.143690 11274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:19:59.144078 11274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:19:59.145234 11274 hybrid_clock.cc:648] HybridClock initialized: now 1790626799145220 us; error 32 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T20:19:59Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1306 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:19:59 dist-test-slave-b83v krb5kdc[11287](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:19:59 dist-test-slave-b83v krb5kdc[11287](info): set up 2 sockets
Sep 28 20:19:59 dist-test-slave-b83v krb5kdc[11287](info): commencing operation
krb5kdc: starting...
W20260928 20:20:00.220919 3144 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Sep 28 20:20:00 dist-test-slave-b83v krb5kdc[11287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:20:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:20:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.18.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab.
Entry for principal kudu/127.3.18.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab.
I20260928 20:20:00.351917 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:36865
--webserver_interface=127.3.18.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:40923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.18.62:36865
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:20:00.438460 11300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:20:00.438750 11300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:20:00.438793 11300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:20:00.440120 11300 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:20:00.440203 11300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:20:00.440229 11300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:20:00.440250 11300 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:20:00.440281 11300 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:20:00.441896 11300 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:40923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.18.62:36865
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:36865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.18.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:20:00.442227 11300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:20:00.442481 11300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:20:00.448017 11308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:20:00.448030 11306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:20:00.448000 11305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:20:00.448264 11300 server_base.cc:1034] running on GCE node
I20260928 20:20:00.448616 11300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:20:00.448956 11300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:20:00.450160 11300 hybrid_clock.cc:648] HybridClock initialized: now 1790626800450104 us; error 75 us; skew 500 ppm
Sep 28 20:20:00 dist-test-slave-b83v krb5kdc[11287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626800, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:20:00.458961 11300 init.cc:377] Logged in from keytab as kudu/127.3.18.62@KRBTEST.COM (short username kudu)
I20260928 20:20:00.459882 11300 webserver.cc:460] Webserver started at http://127.3.18.62:41873/ using document root <none> and password file <none>
I20260928 20:20:00.460197 11300 fs_manager.cc:359] Metadata directory not provided
I20260928 20:20:00.460280 11300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:20:00.460441 11300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:20:00.461745 11300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/master-0/data/instance:
uuid: "771581be3103491a9a655f26d2e4a1c0"
format_stamp: "Formatted at 2026-09-28 20:20:00 on dist-test-slave-b83v"
server_key: "4768bcd3b42ec5e632e519c43de64a8a"
server_key_iv: "f3cdf8dca96502349a391921b8536fca"
server_key_version: "encryptionkey@0"
I20260928 20:20:00.462325 11300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/master-0/wal/instance:
uuid: "771581be3103491a9a655f26d2e4a1c0"
format_stamp: "Formatted at 2026-09-28 20:20:00 on dist-test-slave-b83v"
server_key: "4768bcd3b42ec5e632e519c43de64a8a"
server_key_iv: "f3cdf8dca96502349a391921b8536fca"
server_key_version: "encryptionkey@0"
I20260928 20:20:00.464085 11300 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260928 20:20:00.465117 11315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:20:00.465462 11300 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:20:00.465616 11300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/master-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/master-0/wal
uuid: "771581be3103491a9a655f26d2e4a1c0"
format_stamp: "Formatted at 2026-09-28 20:20:00 on dist-test-slave-b83v"
server_key: "4768bcd3b42ec5e632e519c43de64a8a"
server_key_iv: "f3cdf8dca96502349a391921b8536fca"
server_key_version: "encryptionkey@0"
I20260928 20:20:00.465718 11300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:20:00.478556 11300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:20:00.481106 11300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:20:00.481314 11300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:20:00.485996 11300 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.62:36865
I20260928 20:20:00.486033 11367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.62:36865 every 8 connection(s)
I20260928 20:20:00.486456 11300 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
I20260928 20:20:00.487438 11368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:20:00.487684 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 11300
I20260928 20:20:00.487834 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/master-0/wal/instance
I20260928 20:20:00.488050 3144 external_mini_cluster.cc:1428] Setting key 6d4296f99e04efcc18cf33ee17cc60a0
I20260928 20:20:00.490236 11368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 771581be3103491a9a655f26d2e4a1c0: Bootstrap starting.
I20260928 20:20:00.490844 11368 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 771581be3103491a9a655f26d2e4a1c0: Neither blocks nor log segments found. Creating new log.
I20260928 20:20:00.491151 11368 log.cc:824] T 00000000000000000000000000000000 P 771581be3103491a9a655f26d2e4a1c0: Log is configured to *not* fsync() on all Append() calls
I20260928 20:20:00.492012 11368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 771581be3103491a9a655f26d2e4a1c0: No bootstrap required, opened a new log
I20260928 20:20:00.494023 11368 raft_consensus.cc:348] T 00000000000000000000000000000000 P 771581be3103491a9a655f26d2e4a1c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "771581be3103491a9a655f26d2e4a1c0" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36865 } }
I20260928 20:20:00.494238 11368 raft_consensus.cc:374] T 00000000000000000000000000000000 P 771581be3103491a9a655f26d2e4a1c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:20:00.494297 11368 raft_consensus.cc:729] T 00000000000000000000000000000000 P 771581be3103491a9a655f26d2e4a1c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 771581be3103491a9a655f26d2e4a1c0, State: Initialized, Role: FOLLOWER
I20260928 20:20:00.494457 11368 consensus_queue.cc:260] T 00000000000000000000000000000000 P 771581be3103491a9a655f26d2e4a1c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "771581be3103491a9a655f26d2e4a1c0" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36865 } }
I20260928 20:20:00.494556 11368 raft_consensus.cc:388] T 00000000000000000000000000000000 P 771581be3103491a9a655f26d2e4a1c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:20:00.494609 11368 raft_consensus.cc:482] T 00000000000000000000000000000000 P 771581be3103491a9a655f26d2e4a1c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:20:00.494658 11368 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 771581be3103491a9a655f26d2e4a1c0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:20:00.495253 11368 raft_consensus.cc:504] T 00000000000000000000000000000000 P 771581be3103491a9a655f26d2e4a1c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "771581be3103491a9a655f26d2e4a1c0" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36865 } }
I20260928 20:20:00.495366 11368 leader_election.cc:302] T 00000000000000000000000000000000 P 771581be3103491a9a655f26d2e4a1c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 771581be3103491a9a655f26d2e4a1c0; no voters:
I20260928 20:20:00.495553 11368 leader_election.cc:288] T 00000000000000000000000000000000 P 771581be3103491a9a655f26d2e4a1c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:20:00.495615 11373 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 771581be3103491a9a655f26d2e4a1c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:20:00.495838 11373 raft_consensus.cc:686] T 00000000000000000000000000000000 P 771581be3103491a9a655f26d2e4a1c0 [term 1 LEADER]: Becoming Leader. State: Replica: 771581be3103491a9a655f26d2e4a1c0, State: Running, Role: LEADER
I20260928 20:20:00.495867 11368 sys_catalog.cc:564] T 00000000000000000000000000000000 P 771581be3103491a9a655f26d2e4a1c0 [sys.catalog]: configured and running, proceeding with master startup.
Sep 28 20:20:00 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:20:00.496728 11373 consensus_queue.cc:237] T 00000000000000000000000000000000 P 771581be3103491a9a655f26d2e4a1c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "771581be3103491a9a655f26d2e4a1c0" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36865 } }
I20260928 20:20:00.497485 11375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 771581be3103491a9a655f26d2e4a1c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "771581be3103491a9a655f26d2e4a1c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "771581be3103491a9a655f26d2e4a1c0" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36865 } } }
I20260928 20:20:00.497634 11374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 771581be3103491a9a655f26d2e4a1c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 771581be3103491a9a655f26d2e4a1c0. Latest consensus state: current_term: 1 leader_uuid: "771581be3103491a9a655f26d2e4a1c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "771581be3103491a9a655f26d2e4a1c0" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36865 } } }
I20260928 20:20:00.497814 11375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 771581be3103491a9a655f26d2e4a1c0 [sys.catalog]: This master's current role is: LEADER
I20260928 20:20:00.497828 11374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 771581be3103491a9a655f26d2e4a1c0 [sys.catalog]: This master's current role is: LEADER
W20260928 20:20:00.498173 11387 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 771581be3103491a9a655f26d2e4a1c0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 20:20:00.498301 11387 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 20:20:00.498402 11388 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:20:00.498679 11388 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:20:00.500463 11388 catalog_manager.cc:1347] Generated new cluster ID: 93882511ebcf4c34ac85719a5385e1e7
I20260928 20:20:00.500525 11388 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:20:00.501049 11371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:00.488720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60818 (local address 127.3.18.62:36865)
0928 20:20:00.488928 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:00.488941 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:00.488981 (+ 40us) server_negotiation.cc:407] Connection header received
0928 20:20:00.489470 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:00.489478 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:00.489532 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:00.489752 (+ 220us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:00.490513 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:00.491888 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:00.492856 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:00.493361 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:00.493522 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:00.496557 (+ 3035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:20:00.496562 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:20:00.496563 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:20:00.496578 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:20:00.498864 (+ 2286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:20:00.499308 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:20:00.499311 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:20:00.499311 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:20:00.499417 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:20:00.499659 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:20:00.499661 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:20:00.499661 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:20:00.500047 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:20:00.500238 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0928 20:20:00.500694 (+ 456us) server_negotiation.cc:299] Negotiation successful
0928 20:20:00.500831 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":86,"threads_started":1}
I20260928 20:20:00.513774 11388 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:20:00.514338 11388 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:20:00.519412 11388 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 771581be3103491a9a655f26d2e4a1c0: Generated new TSK 0
I20260928 20:20:00.519618 11388 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.18.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab.
Entry for principal kudu/127.3.18.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab.
I20260928 20:20:00.551617 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.1:0
--local_ip_for_outbound_sockets=127.3.18.1
--webserver_interface=127.3.18.1
--webserver_port=0
--tserver_master_addrs=127.3.18.62:36865
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab
--principal=kudu/127.3.18.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:40923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:20:00.656599 11394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:20:00.656908 11394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:20:00.656953 11394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:20:00.658708 11394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:20:00.658805 11394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:20:00.658859 11394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.1
I20260928 20:20:00.661015 11394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:40923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab
--principal=kudu/127.3.18.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.18.1
--webserver_port=0
--tserver_master_addrs=127.3.18.62:36865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.1
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:20:00.661332 11394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:20:00.661674 11394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:20:00.667678 11399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:20:00.667623 11400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:20:00.667623 11402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:20:00.667925 11394 server_base.cc:1034] running on GCE node
I20260928 20:20:00.668133 11394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:20:00.668344 11394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:20:00.669512 11394 hybrid_clock.cc:648] HybridClock initialized: now 1790626800669498 us; error 32 us; skew 500 ppm
Sep 28 20:20:00 dist-test-slave-b83v krb5kdc[11287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626800, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:20:00.672322 11394 init.cc:377] Logged in from keytab as kudu/127.3.18.1@KRBTEST.COM (short username kudu)
I20260928 20:20:00.672899 11394 webserver.cc:460] Webserver started at http://127.3.18.1:36633/ using document root <none> and password file <none>
I20260928 20:20:00.673166 11394 fs_manager.cc:359] Metadata directory not provided
I20260928 20:20:00.673215 11394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:20:00.673352 11394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:20:00.674475 11394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-0/data/instance:
uuid: "c011d01ade4a48a888653e2a84be7c13"
format_stamp: "Formatted at 2026-09-28 20:20:00 on dist-test-slave-b83v"
server_key: "954729cddd9b686fe522ef29c50b9d9e"
server_key_iv: "7ce650e2fb580ccec9ce6f78789caa58"
server_key_version: "encryptionkey@0"
I20260928 20:20:00.674845 11394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-0/wal/instance:
uuid: "c011d01ade4a48a888653e2a84be7c13"
format_stamp: "Formatted at 2026-09-28 20:20:00 on dist-test-slave-b83v"
server_key: "954729cddd9b686fe522ef29c50b9d9e"
server_key_iv: "7ce650e2fb580ccec9ce6f78789caa58"
server_key_version: "encryptionkey@0"
I20260928 20:20:00.676442 11394 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:20:00.677455 11409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:20:00.677672 11394 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:20:00.677793 11394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-0/wal
uuid: "c011d01ade4a48a888653e2a84be7c13"
format_stamp: "Formatted at 2026-09-28 20:20:00 on dist-test-slave-b83v"
server_key: "954729cddd9b686fe522ef29c50b9d9e"
server_key_iv: "7ce650e2fb580ccec9ce6f78789caa58"
server_key_version: "encryptionkey@0"
I20260928 20:20:00.677891 11394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:20:00.691272 11394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:20:00.693645 11394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:20:00.693843 11394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:20:00.694130 11394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:20:00.694481 11394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:20:00.694545 11394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:20:00.694592 11394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:20:00.694648 11394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:20:00.701470 11394 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.1:36621
I20260928 20:20:00.701535 11522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.1:36621 every 8 connection(s)
I20260928 20:20:00.701908 11394 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
I20260928 20:20:00.707653 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 11394
I20260928 20:20:00.707818 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-0/wal/instance
I20260928 20:20:00.708047 3144 external_mini_cluster.cc:1428] Setting key bf6d03e7f7b14245cf08c503ef21b7b4
Sep 28 20:20:00 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626800, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.1@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:20:00.712756 11371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:00.702736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:48453 (local address 127.3.18.62:36865)
0928 20:20:00.702796 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:00.702798 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:00.702974 (+ 176us) server_negotiation.cc:407] Connection header received
0928 20:20:00.703530 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:00.703531 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:00.703543 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:00.703604 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:00.704576 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:00.705086 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:00.706143 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:00.706356 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:00.706420 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:00.709153 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:20:00.709154 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:20:00.709154 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:20:00.709161 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:20:00.710989 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:20:00.711440 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:20:00.711441 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:20:00.711441 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:20:00.711506 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:20:00.711892 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:20:00.711893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:20:00.711893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:20:00.712054 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:20:00.712134 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 20:20:00.712496 (+ 362us) server_negotiation.cc:299] Negotiation successful
0928 20:20:00.712624 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 20:20:00.712885 11525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:00.702722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36865 (local address 127.3.18.1:48453)
0928 20:20:00.702866 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:00.702902 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 20:20:00.703428 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:20:00.703662 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:20:00.703666 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:20:00.704283 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:20:00.704483 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:00.704488 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:00.705174 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:00.705175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:00.706004 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:00.706007 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:00.706444 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:00.706445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:00.706554 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:00.707119 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:20:00.707129 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:20:00.709056 (+ 1927us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:20:00.711095 (+ 2039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:20:00.711101 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:20:00.711103 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:20:00.711357 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:20:00.711621 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:20:00.711622 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:20:00.711623 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:20:00.711747 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:20:00.712160 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:20:00.712167 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:20:00.712327 (+ 160us) client_negotiation.cc:769] Sending connection context
0928 20:20:00.712424 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 20:20:00.712625 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":79,"threads_started":1}
I20260928 20:20:00.713439 11523 heartbeater.cc:347] Connected to a master server at 127.3.18.62:36865
I20260928 20:20:00.713564 11523 heartbeater.cc:464] Registering TS with master...
I20260928 20:20:00.713805 11523 heartbeater.cc:511] Master 127.3.18.62:36865 requested a full tablet report, sending...
I20260928 20:20:00.714381 11332 ts_manager.cc:194] Registered new tserver with Master: c011d01ade4a48a888653e2a84be7c13 (127.3.18.1:36621)
I20260928 20:20:00.715167 11332 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.18.1@KRBTEST.COM'} at 127.3.18.1:48453
WARNING: no policy specified for kudu/127.3.18.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab.
Entry for principal kudu/127.3.18.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab.
I20260928 20:20:00.739172 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.2:0
--local_ip_for_outbound_sockets=127.3.18.2
--webserver_interface=127.3.18.2
--webserver_port=0
--tserver_master_addrs=127.3.18.62:36865
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab
--principal=kudu/127.3.18.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:40923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:20:00.824332 11528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:20:00.824565 11528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:20:00.824591 11528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:20:00.825788 11528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:20:00.825825 11528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:20:00.825853 11528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.2
I20260928 20:20:00.827345 11528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:40923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab
--principal=kudu/127.3.18.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.18.2
--webserver_port=0
--tserver_master_addrs=127.3.18.62:36865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.2
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:20:00.827553 11528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:20:00.827747 11528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 20:20:00.833194 11528 server_base.cc:1034] running on GCE node
W20260928 20:20:00.833145 11533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:20:00.833127 11534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:20:00.833160 11536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:20:00.833669 11528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:20:00.833953 11528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:20:00.835150 11528 hybrid_clock.cc:648] HybridClock initialized: now 1790626800835160 us; error 63 us; skew 500 ppm
Sep 28 20:20:00 dist-test-slave-b83v krb5kdc[11287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626800, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:20:00.837808 11528 init.cc:377] Logged in from keytab as kudu/127.3.18.2@KRBTEST.COM (short username kudu)
I20260928 20:20:00.838335 11528 webserver.cc:460] Webserver started at http://127.3.18.2:37747/ using document root <none> and password file <none>
I20260928 20:20:00.838551 11528 fs_manager.cc:359] Metadata directory not provided
I20260928 20:20:00.838593 11528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:20:00.838691 11528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:20:00.839748 11528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-1/data/instance:
uuid: "dcf13d048ebf4d0e884e7801a6ed546b"
format_stamp: "Formatted at 2026-09-28 20:20:00 on dist-test-slave-b83v"
server_key: "1b6b67402c84cc8c14ded83c859340d5"
server_key_iv: "b50bf8f1268a03d94e724508439ab37e"
server_key_version: "encryptionkey@0"
I20260928 20:20:00.840170 11528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-1/wal/instance:
uuid: "dcf13d048ebf4d0e884e7801a6ed546b"
format_stamp: "Formatted at 2026-09-28 20:20:00 on dist-test-slave-b83v"
server_key: "1b6b67402c84cc8c14ded83c859340d5"
server_key_iv: "b50bf8f1268a03d94e724508439ab37e"
server_key_version: "encryptionkey@0"
I20260928 20:20:00.841622 11528 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:20:00.842391 11543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:20:00.842638 11528 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:20:00.842731 11528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-1/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-1/wal
uuid: "dcf13d048ebf4d0e884e7801a6ed546b"
format_stamp: "Formatted at 2026-09-28 20:20:00 on dist-test-slave-b83v"
server_key: "1b6b67402c84cc8c14ded83c859340d5"
server_key_iv: "b50bf8f1268a03d94e724508439ab37e"
server_key_version: "encryptionkey@0"
I20260928 20:20:00.842792 11528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:20:00.854157 11528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:20:00.856503 11528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:20:00.856649 11528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:20:00.856859 11528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:20:00.857209 11528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:20:00.857270 11528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:20:00.857317 11528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:20:00.857344 11528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:20:00.863184 11528 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.2:36133
I20260928 20:20:00.863266 11656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.2:36133 every 8 connection(s)
I20260928 20:20:00.863750 11528 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
I20260928 20:20:00.864387 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 11528
I20260928 20:20:00.864513 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-1/wal/instance
I20260928 20:20:00.864813 3144 external_mini_cluster.cc:1428] Setting key 31414d6a06aee6a63ef4f216afb96aff
Sep 28 20:20:00 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626800, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.2@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.18.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.3@KRBTEST.COM" created.
I20260928 20:20:00.877271 11371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:00.864765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:57447 (local address 127.3.18.62:36865)
0928 20:20:00.864839 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:00.864841 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:00.865062 (+ 221us) server_negotiation.cc:407] Connection header received
0928 20:20:00.865769 (+ 707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:00.865770 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:00.865781 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:00.865846 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:00.867270 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:00.868220 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:00.869398 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:00.869727 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:00.869854 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:00.873576 (+ 3722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:20:00.873578 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:20:00.873579 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:20:00.873587 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:20:00.875503 (+ 1916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:20:00.876124 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:20:00.876126 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:20:00.876126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:20:00.876195 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:20:00.876447 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:20:00.876448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:20:00.876448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:20:00.876656 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:20:00.876748 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 20:20:00.877119 (+ 371us) server_negotiation.cc:299] Negotiation successful
0928 20:20:00.877200 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 20:20:00.877650 11659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:00.864750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36865 (local address 127.3.18.2:57447)
0928 20:20:00.864919 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:00.864962 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 20:20:00.865653 (+ 691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:20:00.865949 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:20:00.865953 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:20:00.866822 (+ 869us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:20:00.867129 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:00.867138 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:00.868324 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:00.868342 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:00.869285 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:00.869289 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:00.869859 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:00.869860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:00.870017 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:00.870617 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:20:00.870626 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:20:00.873440 (+ 2814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:20:00.875643 (+ 2203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:20:00.875652 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:20:00.875655 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:20:00.876020 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:20:00.876265 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:20:00.876266 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:20:00.876266 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:20:00.876381 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:20:00.876782 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:20:00.876789 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:20:00.876937 (+ 148us) client_negotiation.cc:769] Sending connection context
0928 20:20:00.877080 (+ 143us) client_negotiation.cc:240] Negotiation successful
0928 20:20:00.877288 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":91,"threads_started":1}
I20260928 20:20:00.878242 11657 heartbeater.cc:347] Connected to a master server at 127.3.18.62:36865
I20260928 20:20:00.878367 11657 heartbeater.cc:464] Registering TS with master...
I20260928 20:20:00.878595 11657 heartbeater.cc:511] Master 127.3.18.62:36865 requested a full tablet report, sending...
I20260928 20:20:00.879051 11332 ts_manager.cc:194] Registered new tserver with Master: dcf13d048ebf4d0e884e7801a6ed546b (127.3.18.2:36133)
I20260928 20:20:00.879438 11332 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.18.2@KRBTEST.COM'} at 127.3.18.2:57447
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab.
Entry for principal kudu/127.3.18.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab.
I20260928 20:20:00.896802 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.3:0
--local_ip_for_outbound_sockets=127.3.18.3
--webserver_interface=127.3.18.3
--webserver_port=0
--tserver_master_addrs=127.3.18.62:36865
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab
--principal=kudu/127.3.18.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:40923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:20:00.984593 11662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:20:00.984808 11662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:20:00.984833 11662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:20:00.985998 11662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:20:00.986040 11662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:20:00.986076 11662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.3
I20260928 20:20:00.987555 11662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:40923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab
--principal=kudu/127.3.18.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.18.3
--webserver_port=0
--tserver_master_addrs=127.3.18.62:36865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.3
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:20:00.987744 11662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:20:00.988001 11662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:20:00.993414 11670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:20:00.993402 11667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:20:00.993402 11668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:20:00.993757 11662 server_base.cc:1034] running on GCE node
I20260928 20:20:00.994027 11662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:20:00.994396 11662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:20:00.995616 11662 hybrid_clock.cc:648] HybridClock initialized: now 1790626800995600 us; error 33 us; skew 500 ppm
Sep 28 20:20:00 dist-test-slave-b83v krb5kdc[11287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626800, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:20:00.998649 11662 init.cc:377] Logged in from keytab as kudu/127.3.18.3@KRBTEST.COM (short username kudu)
I20260928 20:20:00.999122 11662 webserver.cc:460] Webserver started at http://127.3.18.3:35923/ using document root <none> and password file <none>
I20260928 20:20:00.999357 11662 fs_manager.cc:359] Metadata directory not provided
I20260928 20:20:00.999423 11662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:20:00.999572 11662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:20:01.000825 11662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-2/data/instance:
uuid: "beca31dea791452ea9ad8c569debd3cc"
format_stamp: "Formatted at 2026-09-28 20:20:00 on dist-test-slave-b83v"
server_key: "4416d2cff8a6840852397699074044c7"
server_key_iv: "9e1be5db74008cbcab2a5e808b2d5a83"
server_key_version: "encryptionkey@0"
I20260928 20:20:01.001269 11662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-2/wal/instance:
uuid: "beca31dea791452ea9ad8c569debd3cc"
format_stamp: "Formatted at 2026-09-28 20:20:00 on dist-test-slave-b83v"
server_key: "4416d2cff8a6840852397699074044c7"
server_key_iv: "9e1be5db74008cbcab2a5e808b2d5a83"
server_key_version: "encryptionkey@0"
I20260928 20:20:01.002797 11662 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 20:20:01.004031 11677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:20:01.004341 11662 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260928 20:20:01.004470 11662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-2/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-2/wal
uuid: "beca31dea791452ea9ad8c569debd3cc"
format_stamp: "Formatted at 2026-09-28 20:20:00 on dist-test-slave-b83v"
server_key: "4416d2cff8a6840852397699074044c7"
server_key_iv: "9e1be5db74008cbcab2a5e808b2d5a83"
server_key_version: "encryptionkey@0"
I20260928 20:20:01.004613 11662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:20:01.034053 11662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:20:01.037662 11662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:20:01.037873 11662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:20:01.038189 11662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:20:01.038547 11662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:20:01.038605 11662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:20:01.038647 11662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:20:01.038677 11662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:20:01.045147 11662 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.3:35969
I20260928 20:20:01.045418 11790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.3:35969 every 8 connection(s)
I20260928 20:20:01.045609 11662 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
Sep 28 20:20:01 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626800, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.3@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:20:01.052791 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 11662
I20260928 20:20:01.052913 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-2/wal/instance
I20260928 20:20:01.053158 3144 external_mini_cluster.cc:1428] Setting key 6e3cf8e5d28cae2278135cb32d6a6eed
I20260928 20:20:01.056680 11371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:01.046469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:53121 (local address 127.3.18.62:36865)
0928 20:20:01.046550 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:01.046552 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:01.046762 (+ 210us) server_negotiation.cc:407] Connection header received
0928 20:20:01.047321 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:01.047322 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:01.047336 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:01.047387 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:01.048452 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:01.048959 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:01.050022 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:01.050221 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:01.050307 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:01.053127 (+ 2820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:20:01.053129 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:20:01.053129 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:20:01.053138 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:20:01.055166 (+ 2028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:20:01.055644 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:20:01.055646 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:20:01.055646 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:20:01.055714 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:20:01.056036 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:20:01.056037 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:20:01.056038 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:20:01.056174 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:20:01.056234 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 20:20:01.056533 (+ 299us) server_negotiation.cc:299] Negotiation successful
0928 20:20:01.056612 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 20:20:01.056957 11793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:01.046443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36865 (local address 127.3.18.3:53121)
0928 20:20:01.046616 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:01.046661 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 20:20:01.047203 (+ 542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:20:01.047463 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:20:01.047467 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:20:01.048136 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:20:01.048341 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:01.048347 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:01.049074 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:01.049076 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:01.049903 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:01.049905 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:01.050320 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:01.050321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:01.050458 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:01.051083 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:20:01.051092 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:20:01.052989 (+ 1897us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:20:01.055257 (+ 2268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:20:01.055266 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:20:01.055268 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:20:01.055542 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:20:01.055836 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:20:01.055837 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:20:01.055837 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:20:01.055969 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:20:01.056249 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:20:01.056255 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:20:01.056398 (+ 143us) client_negotiation.cc:769] Sending connection context
0928 20:20:01.056500 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 20:20:01.056680 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":87,"threads_started":1}
I20260928 20:20:01.057559 11791 heartbeater.cc:347] Connected to a master server at 127.3.18.62:36865
I20260928 20:20:01.057711 11791 heartbeater.cc:464] Registering TS with master...
I20260928 20:20:01.057953 11791 heartbeater.cc:511] Master 127.3.18.62:36865 requested a full tablet report, sending...
I20260928 20:20:01.058404 11332 ts_manager.cc:194] Registered new tserver with Master: beca31dea791452ea9ad8c569debd3cc (127.3.18.3:35969)
I20260928 20:20:01.058842 11332 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.18.3@KRBTEST.COM'} at 127.3.18.3:53121
I20260928 20:20:01.065757 3144 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.3.18.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.18.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.18.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/krb5kdc/oryx_127.3.18.1.keytab.
Entry for principal oryx/127.3.18.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/krb5kdc/oryx_127.3.18.1.keytab.
WARNING: no policy specified for oryx/127.3.18.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.18.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.18.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/krb5kdc/oryx_127.3.18.2.keytab.
Entry for principal oryx/127.3.18.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/krb5kdc/oryx_127.3.18.2.keytab.
WARNING: no policy specified for oryx/127.3.18.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.18.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.18.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/krb5kdc/oryx_127.3.18.3.keytab.
Entry for principal oryx/127.3.18.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/krb5kdc/oryx_127.3.18.3.keytab.
I20260928 20:20:01.147541 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 11394
I20260928 20:20:01.152765 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 11528
I20260928 20:20:01.157847 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 11662
I20260928 20:20:01.162598 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 11300
I20260928 20:20:01.168152 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:36865
--webserver_interface=127.3.18.62
--webserver_port=41873
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:40923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.18.62:36865
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:20:01.256611 11800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:20:01.256930 11800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:20:01.256989 11800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:20:01.258445 11800 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:20:01.258512 11800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:20:01.258528 11800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:20:01.258541 11800 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:20:01.258553 11800 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:20:01.260236 11800 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:40923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.18.62:36865
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:36865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.18.62
--webserver_port=41873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:20:01.260469 11800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:20:01.260679 11800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:20:01.266026 11806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:20:01.266026 11808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:20:01.266101 11805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:20:01.266330 11800 server_base.cc:1034] running on GCE node
I20260928 20:20:01.266641 11800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:20:01.266999 11800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:20:01.268183 11800 hybrid_clock.cc:648] HybridClock initialized: now 1790626801268154 us; error 65 us; skew 500 ppm
Sep 28 20:20:01 dist-test-slave-b83v krb5kdc[11287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626801, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:20:01.270963 11800 init.cc:377] Logged in from keytab as kudu/127.3.18.62@KRBTEST.COM (short username kudu)
I20260928 20:20:01.271533 11800 webserver.cc:460] Webserver started at http://127.3.18.62:41873/ using document root <none> and password file <none>
I20260928 20:20:01.271842 11800 fs_manager.cc:359] Metadata directory not provided
I20260928 20:20:01.271932 11800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:20:01.273442 11800 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:20:01.274243 11815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:20:01.274452 11800 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:20:01.274569 11800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/master-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/master-0/wal
uuid: "771581be3103491a9a655f26d2e4a1c0"
format_stamp: "Formatted at 2026-09-28 20:20:00 on dist-test-slave-b83v"
server_key: "4768bcd3b42ec5e632e519c43de64a8a"
server_key_iv: "f3cdf8dca96502349a391921b8536fca"
server_key_version: "encryptionkey@0"
I20260928 20:20:01.274968 11800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:20:01.287551 11800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:20:01.290125 11800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:20:01.290333 11800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:20:01.294831 11800 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.62:36865
I20260928 20:20:01.294866 11867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.62:36865 every 8 connection(s)
I20260928 20:20:01.295295 11800 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
I20260928 20:20:01.296868 11868 sys_catalog.cc:263] Verifying existing consensus state
I20260928 20:20:01.297564 11868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 771581be3103491a9a655f26d2e4a1c0: Bootstrap starting.
I20260928 20:20:01.303828 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 11800
I20260928 20:20:01.304172 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.1:36621
--local_ip_for_outbound_sockets=127.3.18.1
--tserver_master_addrs=127.3.18.62:36865
--webserver_port=36633
--webserver_interface=127.3.18.1
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab
--principal=kudu/127.3.18.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:40923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.18.1
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/krb5kdc/oryx_127.3.18.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 20:20:01.305963 11868 log.cc:824] T 00000000000000000000000000000000 P 771581be3103491a9a655f26d2e4a1c0: Log is configured to *not* fsync() on all Append() calls
I20260928 20:20:01.308512 11868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 771581be3103491a9a655f26d2e4a1c0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 20:20:01.308879 11868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 771581be3103491a9a655f26d2e4a1c0: Bootstrap complete.
I20260928 20:20:01.313028 11868 raft_consensus.cc:348] T 00000000000000000000000000000000 P 771581be3103491a9a655f26d2e4a1c0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "771581be3103491a9a655f26d2e4a1c0" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36865 } }
I20260928 20:20:01.313459 11868 raft_consensus.cc:729] T 00000000000000000000000000000000 P 771581be3103491a9a655f26d2e4a1c0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 771581be3103491a9a655f26d2e4a1c0, State: Initialized, Role: FOLLOWER
I20260928 20:20:01.313671 11868 consensus_queue.cc:260] T 00000000000000000000000000000000 P 771581be3103491a9a655f26d2e4a1c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "771581be3103491a9a655f26d2e4a1c0" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36865 } }
I20260928 20:20:01.313777 11868 raft_consensus.cc:388] T 00000000000000000000000000000000 P 771581be3103491a9a655f26d2e4a1c0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:20:01.313843 11868 raft_consensus.cc:482] T 00000000000000000000000000000000 P 771581be3103491a9a655f26d2e4a1c0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:20:01.313906 11868 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 771581be3103491a9a655f26d2e4a1c0 [term 1 FOLLOWER]: Advancing to term 2
I20260928 20:20:01.315639 11868 raft_consensus.cc:504] T 00000000000000000000000000000000 P 771581be3103491a9a655f26d2e4a1c0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "771581be3103491a9a655f26d2e4a1c0" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36865 } }
I20260928 20:20:01.315806 11868 leader_election.cc:302] T 00000000000000000000000000000000 P 771581be3103491a9a655f26d2e4a1c0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 771581be3103491a9a655f26d2e4a1c0; no voters:
I20260928 20:20:01.316009 11868 leader_election.cc:288] T 00000000000000000000000000000000 P 771581be3103491a9a655f26d2e4a1c0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 20:20:01.316109 11872 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 771581be3103491a9a655f26d2e4a1c0 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 20:20:01.316314 11868 sys_catalog.cc:564] T 00000000000000000000000000000000 P 771581be3103491a9a655f26d2e4a1c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:20:01.316313 11872 raft_consensus.cc:686] T 00000000000000000000000000000000 P 771581be3103491a9a655f26d2e4a1c0 [term 2 LEADER]: Becoming Leader. State: Replica: 771581be3103491a9a655f26d2e4a1c0, State: Running, Role: LEADER
I20260928 20:20:01.316493 11872 consensus_queue.cc:237] T 00000000000000000000000000000000 P 771581be3103491a9a655f26d2e4a1c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "771581be3103491a9a655f26d2e4a1c0" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36865 } }
I20260928 20:20:01.316955 11874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 771581be3103491a9a655f26d2e4a1c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 771581be3103491a9a655f26d2e4a1c0. Latest consensus state: current_term: 2 leader_uuid: "771581be3103491a9a655f26d2e4a1c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "771581be3103491a9a655f26d2e4a1c0" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36865 } } }
I20260928 20:20:01.317052 11874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 771581be3103491a9a655f26d2e4a1c0 [sys.catalog]: This master's current role is: LEADER
I20260928 20:20:01.317327 11881 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:20:01.316942 11873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 771581be3103491a9a655f26d2e4a1c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "771581be3103491a9a655f26d2e4a1c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "771581be3103491a9a655f26d2e4a1c0" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36865 } } }
I20260928 20:20:01.317601 11873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 771581be3103491a9a655f26d2e4a1c0 [sys.catalog]: This master's current role is: LEADER
I20260928 20:20:01.317976 11881 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:20:01.318207 11881 catalog_manager.cc:1259] Loaded cluster ID: 93882511ebcf4c34ac85719a5385e1e7
I20260928 20:20:01.318378 11881 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:20:01.319195 11881 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:20:01.319402 11881 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 771581be3103491a9a655f26d2e4a1c0: Loaded TSK: 0
I20260928 20:20:01.319640 11881 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 20:20:01.401346 11870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:20:01.401572 11870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:20:01.401598 11870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:20:01.402887 11870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:20:01.402937 11870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:20:01.402966 11870 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.1
I20260928 20:20:01.404635 11870 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:40923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/krb5kdc/oryx_127.3.18.1.keytab
--principal=oryx/127.3.18.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.1:36621
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.18.1
--webserver_port=36633
--tserver_master_addrs=127.3.18.62:36865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.1
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:20:01.404927 11870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:20:01.405210 11870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:20:01.410827 11894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:20:01.410826 11897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:20:01.410820 11895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:20:01.411432 11870 server_base.cc:1034] running on GCE node
I20260928 20:20:01.411808 11870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:20:01.412119 11870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:20:01.413321 11870 hybrid_clock.cc:648] HybridClock initialized: now 1790626801413304 us; error 32 us; skew 500 ppm
Sep 28 20:20:01 dist-test-slave-b83v krb5kdc[11287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626801, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.18.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:20:01.416313 11870 init.cc:377] Logged in from keytab as oryx/127.3.18.1@KRBTEST.COM (short username oryx)
I20260928 20:20:01.416855 11870 webserver.cc:460] Webserver started at http://127.3.18.1:36633/ using document root <none> and password file <none>
I20260928 20:20:01.417227 11870 fs_manager.cc:359] Metadata directory not provided
I20260928 20:20:01.417327 11870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:20:01.419003 11870 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260928 20:20:01.419677 11904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:20:01.419888 11870 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:20:01.420009 11870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-0/wal
uuid: "c011d01ade4a48a888653e2a84be7c13"
format_stamp: "Formatted at 2026-09-28 20:20:00 on dist-test-slave-b83v"
server_key: "954729cddd9b686fe522ef29c50b9d9e"
server_key_iv: "7ce650e2fb580ccec9ce6f78789caa58"
server_key_version: "encryptionkey@0"
I20260928 20:20:01.420396 11870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:20:01.436964 11870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:20:01.439700 11870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:20:01.439921 11870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:20:01.440219 11870 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:20:01.440575 11870 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:20:01.440634 11870 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:20:01.440671 11870 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:20:01.440698 11870 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:20:01.447335 11870 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.1:36621
I20260928 20:20:01.447389 12017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.1:36621 every 8 connection(s)
I20260928 20:20:01.447839 11870 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
I20260928 20:20:01.450392 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 11870
I20260928 20:20:01.450711 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.2:36133
--local_ip_for_outbound_sockets=127.3.18.2
--tserver_master_addrs=127.3.18.62:36865
--webserver_port=37747
--webserver_interface=127.3.18.2
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab
--principal=kudu/127.3.18.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:40923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.18.2
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/krb5kdc/oryx_127.3.18.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 20:20:01 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.1@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:20:01 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.1@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:20:01.457798 12020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:01.448712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36865 (local address 127.3.18.1:46185)
0928 20:20:01.448895 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:01.448940 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 20:20:01.449423 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:20:01.449951 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:20:01.449955 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:20:01.450666 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:20:01.450965 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:01.450972 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:01.452381 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:01.452383 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:01.453654 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:01.453658 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:01.454305 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:01.454307 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:01.454444 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:01.455051 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:20:01.455060 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:20:01.457540 (+ 2480us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.18.62:36865: Server oryx/127.3.18.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":101,"threads_started":1}
W20260928 20:20:01.458214 12018 heartbeater.cc:644] Failed to heartbeat to 127.3.18.62:36865 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.18.62:36865: Client connection negotiation failed: client connection to 127.3.18.62:36865: Server oryx/127.3.18.62@KRBTEST.COM not found in Kerberos database
W20260928 20:20:01.458622 12021 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:20:01.448803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:46185 (local address 127.3.18.62:36865)
0928 20:20:01.449013 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:01.449033 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:01.449080 (+ 47us) server_negotiation.cc:407] Connection header received
0928 20:20:01.449647 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:01.449655 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:01.449716 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:01.449877 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:01.451101 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:01.452235 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:01.453754 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:01.454186 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:01.454314 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:01.457710 (+ 3396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.18.1:46185: BlockingRecv error: recv got EOF from 127.3.18.1:46185 (error 108)
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":95,"threads_started":1}
Sep 28 20:20:01 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.1@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:20:01 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.1@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:20:01.463562 12020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:01.458509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36865 (local address 127.3.18.1:53821)
0928 20:20:01.458781 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:01.458793 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:20:01.458839 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:20:01.459071 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:20:01.459072 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:20:01.459247 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:20:01.459301 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:01.459302 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:01.460072 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:01.460073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:01.460593 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:01.460596 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:01.460972 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:01.460973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:01.461052 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:01.461553 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:20:01.461557 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:20:01.463471 (+ 1914us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.18.62:36865: Server oryx/127.3.18.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":253}
W20260928 20:20:01.463629 12021 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:20:01.458563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:53821 (local address 127.3.18.62:36865)
0928 20:20:01.458948 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:01.458950 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:01.458956 (+ 6us) server_negotiation.cc:407] Connection header received
0928 20:20:01.458978 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:01.458979 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:01.458991 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:01.459054 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:01.459387 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:01.459989 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:01.460687 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:01.460896 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:01.460975 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:01.463562 (+ 2587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.18.1:53821: BlockingRecv error: recv got EOF from 127.3.18.1:53821 (error 108)
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":103,"threads_started":1}
Sep 28 20:20:01 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.1@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:20:01 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.1@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:20:01.468603 12020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:01.464167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36865 (local address 127.3.18.1:56905)
0928 20:20:01.464263 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:01.464274 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:20:01.464321 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:20:01.464472 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:20:01.464473 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:20:01.464616 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:20:01.464649 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:01.464650 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:01.465360 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:01.465361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:01.465870 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:01.465871 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:01.466240 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:01.466241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:01.466295 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:01.466754 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:20:01.466760 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:20:01.468502 (+ 1742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.18.62:36865: Server oryx/127.3.18.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":82}
W20260928 20:20:01.468642 12021 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:20:01.464226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:56905 (local address 127.3.18.62:36865)
0928 20:20:01.464269 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:01.464272 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:01.464305 (+ 33us) server_negotiation.cc:407] Connection header received
0928 20:20:01.464374 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:01.464374 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:01.464386 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:01.464455 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:01.464725 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:01.465279 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:01.465957 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:01.466154 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:01.466241 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:01.468580 (+ 2339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.18.1:56905: BlockingRecv error: recv got EOF from 127.3.18.1:56905 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260928 20:20:01.469015 12018 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 20:20:01.538065 12022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:20:01.538326 12022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:20:01.538352 12022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:20:01.539587 12022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:20:01.539629 12022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:20:01.539657 12022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.2
I20260928 20:20:01.541395 12022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:40923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/krb5kdc/oryx_127.3.18.2.keytab
--principal=oryx/127.3.18.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.2:36133
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.18.2
--webserver_port=37747
--tserver_master_addrs=127.3.18.62:36865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.2
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:20:01.541628 12022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:20:01.541829 12022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:20:01.547158 12029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:20:01.547307 12028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:20:01.547163 12031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:20:01.547382 12022 server_base.cc:1034] running on GCE node
I20260928 20:20:01.547744 12022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:20:01.548111 12022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:20:01.549306 12022 hybrid_clock.cc:648] HybridClock initialized: now 1790626801549300 us; error 49 us; skew 500 ppm
Sep 28 20:20:01 dist-test-slave-b83v krb5kdc[11287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626801, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.18.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:20:01.552271 12022 init.cc:377] Logged in from keytab as oryx/127.3.18.2@KRBTEST.COM (short username oryx)
I20260928 20:20:01.552942 12022 webserver.cc:460] Webserver started at http://127.3.18.2:37747/ using document root <none> and password file <none>
I20260928 20:20:01.553218 12022 fs_manager.cc:359] Metadata directory not provided
I20260928 20:20:01.553316 12022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:20:01.554869 12022 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 20:20:01.555626 12038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:20:01.555931 12022 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:20:01.556017 12022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-1/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-1/wal
uuid: "dcf13d048ebf4d0e884e7801a6ed546b"
format_stamp: "Formatted at 2026-09-28 20:20:00 on dist-test-slave-b83v"
server_key: "1b6b67402c84cc8c14ded83c859340d5"
server_key_iv: "b50bf8f1268a03d94e724508439ab37e"
server_key_version: "encryptionkey@0"
I20260928 20:20:01.556432 12022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:20:01.572198 12022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:20:01.574806 12022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:20:01.574939 12022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:20:01.575151 12022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:20:01.575533 12022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:20:01.575609 12022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:20:01.575659 12022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:20:01.575682 12022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:20:01.583578 12022 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.2:36133
I20260928 20:20:01.583652 12151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.2:36133 every 8 connection(s)
I20260928 20:20:01.584118 12022 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
I20260928 20:20:01.587237 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 12022
I20260928 20:20:01.587589 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.3:35969
--local_ip_for_outbound_sockets=127.3.18.3
--tserver_master_addrs=127.3.18.62:36865
--webserver_port=35923
--webserver_interface=127.3.18.3
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab
--principal=kudu/127.3.18.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:40923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.18.3
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/krb5kdc/oryx_127.3.18.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 20:20:01 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.2@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:20:01 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.2@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:20:01.594292 12154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:01.585040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36865 (local address 127.3.18.2:46407)
0928 20:20:01.585213 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:01.585258 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 20:20:01.585767 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:20:01.586068 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:20:01.586072 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:20:01.586734 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:20:01.587029 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:01.587040 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:01.588114 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:01.588115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:01.589452 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:01.589456 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:01.590063 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:01.590065 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:01.590194 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:01.590957 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:20:01.590968 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:20:01.594104 (+ 3136us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.18.62:36865: Server oryx/127.3.18.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":100,"threads_started":1}
W20260928 20:20:01.594362 12021 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:20:01.585074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:46407 (local address 127.3.18.62:36865)
0928 20:20:01.585145 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:01.585148 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:01.585357 (+ 209us) server_negotiation.cc:407] Connection header received
0928 20:20:01.585904 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:01.585906 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:01.585923 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:01.585999 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:01.587209 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:01.587987 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:01.589597 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:01.589933 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:01.590028 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:01.594260 (+ 4232us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.18.2:46407: BlockingRecv error: recv got EOF from 127.3.18.2:46407 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260928 20:20:01.594799 12152 heartbeater.cc:644] Failed to heartbeat to 127.3.18.62:36865 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.18.62:36865: Client connection negotiation failed: client connection to 127.3.18.62:36865: Server oryx/127.3.18.62@KRBTEST.COM not found in Kerberos database
Sep 28 20:20:01 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.2@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:20:01 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.2@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:20:01.600024 12154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:01.595093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36865 (local address 127.3.18.2:53617)
0928 20:20:01.595151 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:01.595160 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 20:20:01.595213 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:20:01.595374 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:20:01.595375 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:20:01.595522 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:20:01.595584 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:01.595586 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:01.596331 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:01.596333 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:01.596850 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:01.596852 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:01.597251 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:01.597252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:01.597328 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:01.597917 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:20:01.597923 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:20:01.599933 (+ 2010us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.18.62:36865: Server oryx/127.3.18.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 20:20:01.600108 12021 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:20:01.595163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:53617 (local address 127.3.18.62:36865)
0928 20:20:01.595237 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:01.595240 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:01.595250 (+ 10us) server_negotiation.cc:407] Connection header received
0928 20:20:01.595284 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:01.595284 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:01.595295 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:01.595370 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:01.595670 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:01.596243 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:01.596943 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:01.597158 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:01.597250 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:01.600029 (+ 2779us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.18.2:53617: BlockingRecv error: recv got EOF from 127.3.18.2:53617 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 28 20:20:01 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.2@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:20:01 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.2@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:20:01.604971 12154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:01.600296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36865 (local address 127.3.18.2:34457)
0928 20:20:01.600417 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:01.600427 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 20:20:01.600475 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:20:01.600631 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:20:01.600632 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:20:01.600764 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:20:01.600800 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:01.600801 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:01.601480 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:01.601490 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:01.602101 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:01.602103 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:01.602473 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:01.602474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:01.602529 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:01.603026 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:20:01.603031 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:20:01.604892 (+ 1861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.18.62:36865: Server oryx/127.3.18.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107}
W20260928 20:20:01.605155 12021 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:20:01.600380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:34457 (local address 127.3.18.62:36865)
0928 20:20:01.600418 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:01.600420 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:01.600460 (+ 40us) server_negotiation.cc:407] Connection header received
0928 20:20:01.600540 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:01.600541 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:01.600550 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:01.600627 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:01.600880 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:01.601379 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:01.602191 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:01.602388 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:01.602463 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:01.605049 (+ 2586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.18.2:34457: BlockingRecv error: recv got EOF from 127.3.18.2:34457 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260928 20:20:01.605157 12152 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 20:20:01.675380 12155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:20:01.675653 12155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:20:01.675697 12155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:20:01.676999 12155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:20:01.677083 12155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:20:01.677136 12155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.3
I20260928 20:20:01.679030 12155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:40923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/krb5kdc/oryx_127.3.18.3.keytab
--principal=oryx/127.3.18.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.3:35969
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.18.3
--webserver_port=35923
--tserver_master_addrs=127.3.18.62:36865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.3
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:20:01.679347 12155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:20:01.679621 12155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:20:01.684845 12160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:20:01.684845 12163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:20:01.684866 12161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:20:01.685217 12155 server_base.cc:1034] running on GCE node
I20260928 20:20:01.685410 12155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:20:01.685627 12155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:20:01.686780 12155 hybrid_clock.cc:648] HybridClock initialized: now 1790626801686768 us; error 34 us; skew 500 ppm
Sep 28 20:20:01 dist-test-slave-b83v krb5kdc[11287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626801, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.18.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:20:01.689469 12155 init.cc:377] Logged in from keytab as oryx/127.3.18.3@KRBTEST.COM (short username oryx)
I20260928 20:20:01.689968 12155 webserver.cc:460] Webserver started at http://127.3.18.3:35923/ using document root <none> and password file <none>
I20260928 20:20:01.690225 12155 fs_manager.cc:359] Metadata directory not provided
I20260928 20:20:01.690284 12155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:20:01.691823 12155 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:20:01.692497 12170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:20:01.692747 12155 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:20:01.692862 12155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-2/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-2/wal
uuid: "beca31dea791452ea9ad8c569debd3cc"
format_stamp: "Formatted at 2026-09-28 20:20:00 on dist-test-slave-b83v"
server_key: "4416d2cff8a6840852397699074044c7"
server_key_iv: "9e1be5db74008cbcab2a5e808b2d5a83"
server_key_version: "encryptionkey@0"
I20260928 20:20:01.693235 12155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:20:01.710018 12155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:20:01.712675 12155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:20:01.712859 12155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:20:01.713114 12155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:20:01.713500 12155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:20:01.713570 12155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:20:01.713622 12155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:20:01.713665 12155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:20:01.720058 12155 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.3:35969
I20260928 20:20:01.720098 12283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.3:35969 every 8 connection(s)
I20260928 20:20:01.720475 12155 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
I20260928 20:20:01.723826 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 12155
Sep 28 20:20:01 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.3@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:20:01 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.3@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 20:20:01.729734 12021 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:20:01.721255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:60183 (local address 127.3.18.62:36865)
0928 20:20:01.721302 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:01.721304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:01.721519 (+ 215us) server_negotiation.cc:407] Connection header received
0928 20:20:01.722128 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:01.722129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:01.722145 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:01.722194 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:01.723305 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:01.723973 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:01.725854 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:01.726020 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:01.726098 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:01.729683 (+ 3585us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.18.3:60183: BlockingRecv error: recv got EOF from 127.3.18.3:60183 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 20:20:01.729803 12286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:01.721226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36865 (local address 127.3.18.3:60183)
0928 20:20:01.721392 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:01.721430 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 20:20:01.721959 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:20:01.722296 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:20:01.722305 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:20:01.722959 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:20:01.723195 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:01.723201 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:01.724077 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:01.724078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:01.725189 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:01.725192 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:01.726117 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:01.726119 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:01.726258 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:01.726810 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:20:01.726821 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:20:01.729563 (+ 2742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.18.62:36865: Server oryx/127.3.18.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":76,"threads_started":1}
W20260928 20:20:01.730054 12284 heartbeater.cc:644] Failed to heartbeat to 127.3.18.62:36865 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.18.62:36865: Client connection negotiation failed: client connection to 127.3.18.62:36865: Server oryx/127.3.18.62@KRBTEST.COM not found in Kerberos database
I20260928 20:20:01.732120 12023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:01.724392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60832 (local address 127.3.18.62:36865)
0928 20:20:01.724453 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:01.724455 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:01.724463 (+ 8us) server_negotiation.cc:407] Connection header received
0928 20:20:01.724537 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:01.724538 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:01.724549 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:01.724626 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:01.725295 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:01.725782 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:01.726572 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:01.726837 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:01.726910 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:01.728348 (+ 1438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:20:01.728355 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:20:01.728355 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:20:01.728370 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:20:01.730447 (+ 2077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:20:01.730863 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:20:01.730866 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:20:01.730866 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:20:01.731006 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:20:01.731239 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:20:01.731240 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:20:01.731240 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:20:01.731544 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:20:01.731693 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 20:20:01.731921 (+ 228us) server_negotiation.cc:299] Negotiation successful
0928 20:20:01.732015 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
Sep 28 20:20:01 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.3@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:20:01 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.3@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:20:01.736316 12286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:01.730333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36865 (local address 127.3.18.3:39017)
0928 20:20:01.730386 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:01.730397 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:20:01.730442 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:20:01.730556 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:20:01.730557 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:20:01.730775 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:20:01.730843 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:01.730846 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:01.731758 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:01.731759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:01.732583 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:01.732586 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:01.733137 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:01.733139 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:01.733246 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:01.733920 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:20:01.733925 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:20:01.736247 (+ 2322us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.18.62:36865: Server oryx/127.3.18.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 20:20:01.736572 12021 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:20:01.730393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:39017 (local address 127.3.18.62:36865)
0928 20:20:01.730434 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:01.730436 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:01.730445 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:20:01.730488 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:01.730489 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:01.730500 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:01.730553 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:01.730914 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:01.731667 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:01.732668 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:01.733010 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:01.733093 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:01.736497 (+ 3404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.18.3:39017: BlockingRecv error: recv got EOF from 127.3.18.3:39017 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 28 20:20:01 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.3@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:20:01 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.3@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:20:01.741228 12286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:01.736764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36865 (local address 127.3.18.3:45205)
0928 20:20:01.736825 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:01.736839 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 20:20:01.736890 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:20:01.737070 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:20:01.737072 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:20:01.737193 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:20:01.737230 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:01.737231 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:01.737910 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:01.737911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:01.738429 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:01.738431 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:01.738756 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:01.738757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:01.738795 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:01.739314 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:20:01.739320 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:20:01.741174 (+ 1854us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.18.62:36865: Server oryx/127.3.18.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 20:20:01.741384 12021 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:20:01.736799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:45205 (local address 127.3.18.62:36865)
0928 20:20:01.736851 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:01.736853 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:01.736865 (+ 12us) server_negotiation.cc:407] Connection header received
0928 20:20:01.736957 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:01.736958 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:01.736966 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:01.737019 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:01.737313 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:01.737812 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:01.738503 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:01.738694 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:01.738742 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:01.741291 (+ 2549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.18.3:45205: BlockingRecv error: recv got EOF from 127.3.18.3:45205 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260928 20:20:01.741448 12284 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 20:20:02 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.1@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:20:02 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.1@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:20:02.482270 12288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:02.469551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36865 (local address 127.3.18.1:50647)
0928 20:20:02.469737 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:02.469749 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:20:02.469819 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:20:02.469984 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:20:02.469985 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:20:02.470203 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:20:02.470260 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:02.470262 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:02.471013 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:02.471015 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:02.471906 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:02.471911 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:02.472457 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:02.472458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:02.472529 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:02.480100 (+ 7571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:20:02.480107 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:20:02.482152 (+ 2045us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.18.62:36865: Server oryx/127.3.18.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":114,"threads_started":1}
W20260928 20:20:02.482316 12289 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:20:02.469696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:50647 (local address 127.3.18.62:36865)
0928 20:20:02.469843 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:02.469845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:02.469855 (+ 10us) server_negotiation.cc:407] Connection header received
0928 20:20:02.469885 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:02.469886 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:02.469896 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:02.469962 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:02.470347 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:02.470910 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:02.472053 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:02.472378 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:02.472444 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:02.482237 (+ 9793us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.18.1:50647: BlockingRecv error: recv got EOF from 127.3.18.1:50647 (error 108)
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":90,"threads_started":1}
Sep 28 20:20:02 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.2@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:20:02 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.2@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:20:02.610656 12290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:02.605727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36865 (local address 127.3.18.2:46017)
0928 20:20:02.605858 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:02.605873 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 20:20:02.605945 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:20:02.606190 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:20:02.606191 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:20:02.606428 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:20:02.606484 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:02.606486 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:02.607212 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:02.607213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:02.607817 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:02.607819 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:02.608264 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:02.608265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:02.608320 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:02.608764 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:20:02.608768 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:20:02.610586 (+ 1818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.18.62:36865: Server oryx/127.3.18.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":66,"threads_started":1}
W20260928 20:20:02.610781 12289 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:20:02.605815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:46017 (local address 127.3.18.62:36865)
0928 20:20:02.605891 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:02.605894 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:02.605928 (+ 34us) server_negotiation.cc:407] Connection header received
0928 20:20:02.606056 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:02.606056 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:02.606068 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:02.606176 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:02.606567 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:02.607059 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:02.607909 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:02.608159 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:02.608233 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:02.610679 (+ 2446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.18.2:46017: BlockingRecv error: recv got EOF from 127.3.18.2:46017 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 28 20:20:02 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.3@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:20:02 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.3@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:20:02.746838 12291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:02.741876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36865 (local address 127.3.18.3:51019)
0928 20:20:02.742096 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:02.742109 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 20:20:02.742186 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:20:02.742336 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:20:02.742337 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:20:02.742520 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:20:02.742568 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:02.742569 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:02.743305 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:02.743306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:02.743907 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:02.743909 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:02.744309 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:02.744311 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:02.744372 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:02.744905 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:20:02.744911 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:20:02.746729 (+ 1818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.18.62:36865: Server oryx/127.3.18.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":109,"threads_started":1}
W20260928 20:20:02.746873 12289 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:20:02.741978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:51019 (local address 127.3.18.62:36865)
0928 20:20:02.742084 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:02.742086 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:02.742167 (+ 81us) server_negotiation.cc:407] Connection header received
0928 20:20:02.742244 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:02.742245 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:02.742259 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:02.742314 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:02.742709 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:02.743216 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:02.744000 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:02.744218 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:02.744308 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:02.746807 (+ 2499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.18.3:51019: BlockingRecv error: recv got EOF from 127.3.18.3:51019 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 28 20:20:03 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.1@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:20:03 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.1@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:20:03.488714 12292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:03.482942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36865 (local address 127.3.18.1:45969)
0928 20:20:03.483188 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:03.483208 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 20:20:03.483277 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:20:03.483528 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:20:03.483529 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:20:03.483742 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:20:03.483877 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:03.483880 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:03.484576 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:03.484578 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:03.485157 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:03.485159 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:03.485530 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:03.485531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:03.485577 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:03.486416 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:20:03.486421 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:20:03.488650 (+ 2229us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.18.62:36865: Server oryx/127.3.18.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":132,"threads_started":1}
W20260928 20:20:03.488888 12293 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:20:03.483112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:45969 (local address 127.3.18.62:36865)
0928 20:20:03.483255 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:03.483258 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:03.483270 (+ 12us) server_negotiation.cc:407] Connection header received
0928 20:20:03.483356 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:03.483356 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:03.483376 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:03.483454 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:03.483969 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:03.484488 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:03.485248 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:03.485450 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:03.485509 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:03.488784 (+ 3275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.18.1:45969: BlockingRecv error: recv got EOF from 127.3.18.1:45969 (error 108)
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":64,"threads_started":1}
Sep 28 20:20:03 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.2@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:20:03 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.2@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:20:03.617389 12294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:03.611263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36865 (local address 127.3.18.2:42705)
0928 20:20:03.611466 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:03.611482 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 20:20:03.611538 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:20:03.611802 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:20:03.611803 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:20:03.611978 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:20:03.612030 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:03.612032 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:03.612902 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:03.612904 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:03.613460 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:03.613462 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:03.614182 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:03.614184 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:03.614256 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:03.614870 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:20:03.614875 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:20:03.617274 (+ 2399us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.18.62:36865: Server oryx/127.3.18.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":95,"threads_started":1}
W20260928 20:20:03.617524 12293 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:20:03.611411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:42705 (local address 127.3.18.62:36865)
0928 20:20:03.611515 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:03.611517 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:03.611640 (+ 123us) server_negotiation.cc:407] Connection header received
0928 20:20:03.611679 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:03.611680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:03.611693 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:03.611826 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:03.612156 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:03.612780 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:03.613667 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:03.614071 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:03.614182 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:03.617434 (+ 3252us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.18.2:42705: BlockingRecv error: recv got EOF from 127.3.18.2:42705 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 28 20:20:03 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.3@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:20:03 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.3@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:20:03.752137 12295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:03.747393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36865 (local address 127.3.18.3:36847)
0928 20:20:03.747576 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:03.747587 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:20:03.747652 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:20:03.747835 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:20:03.747836 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:20:03.747987 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:20:03.748035 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:03.748037 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:03.748721 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:03.748722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:03.749259 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:03.749261 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:03.749641 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:03.749642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:03.749686 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:03.750220 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:20:03.750225 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:20:03.752056 (+ 1831us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.18.62:36865: Server oryx/127.3.18.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":103,"threads_started":1}
W20260928 20:20:03.752269 12293 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:20:03.747533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:36847 (local address 127.3.18.62:36865)
0928 20:20:03.747583 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:03.747585 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:03.747674 (+ 89us) server_negotiation.cc:407] Connection header received
0928 20:20:03.747698 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:03.747698 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:03.747711 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:03.747778 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:03.748116 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:03.748610 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:03.749384 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:03.749569 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:03.749623 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:03.752165 (+ 2542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.18.3:36847: BlockingRecv error: recv got EOF from 127.3.18.3:36847 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 28 20:20:04 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.1@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:20:04 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.1@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:20:04.494715 12296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:04.489221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36865 (local address 127.3.18.1:39655)
0928 20:20:04.489389 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:04.489401 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:20:04.489459 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:20:04.489648 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:20:04.489649 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:20:04.489834 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:20:04.489887 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:04.489889 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:04.490566 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:04.490567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:04.491229 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:04.491232 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:04.491616 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:04.491617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:04.491675 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:04.492309 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:20:04.492314 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:20:04.494589 (+ 2275us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.18.62:36865: Server oryx/127.3.18.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":96,"threads_started":1}
W20260928 20:20:04.494817 12297 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:20:04.489324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:39655 (local address 127.3.18.62:36865)
0928 20:20:04.489457 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:04.489459 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:04.489467 (+ 8us) server_negotiation.cc:407] Connection header received
0928 20:20:04.489512 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:04.489513 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:04.489524 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:04.489692 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:04.489962 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:04.490476 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:04.491324 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:04.491534 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:04.491590 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:04.494719 (+ 3129us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.18.1:39655: BlockingRecv error: recv got EOF from 127.3.18.1:39655 (error 108)
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":60,"threads_started":1}
Sep 28 20:20:04 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.2@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:20:04 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.2@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:20:04.622862 12298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:04.617884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36865 (local address 127.3.18.2:46529)
0928 20:20:04.618057 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:04.618068 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:20:04.618128 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:20:04.618294 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:20:04.618295 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:20:04.618456 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:20:04.618507 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:04.618509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:04.619278 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:04.619279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:04.619930 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:04.619932 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:04.620320 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:04.620321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:04.620371 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:04.620871 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:20:04.620892 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:20:04.622797 (+ 1905us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.18.62:36865: Server oryx/127.3.18.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":90,"threads_started":1}
W20260928 20:20:04.623018 12297 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:20:04.617972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:46529 (local address 127.3.18.62:36865)
0928 20:20:04.618031 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:04.618033 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:04.618113 (+ 80us) server_negotiation.cc:407] Connection header received
0928 20:20:04.618194 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:04.618195 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:04.618210 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:04.618281 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:04.618590 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:04.619188 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:04.620013 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:04.620230 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:04.620293 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:04.622930 (+ 2637us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.18.2:46529: BlockingRecv error: recv got EOF from 127.3.18.2:46529 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 28 20:20:04 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.3@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:20:04 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.3@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:20:04.757887 12299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:04.752738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36865 (local address 127.3.18.3:41291)
0928 20:20:04.752938 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:04.752952 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 20:20:04.753027 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:20:04.753170 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:20:04.753171 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:20:04.753356 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:20:04.753406 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:04.753408 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:04.754124 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:04.754125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:04.754741 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:04.754744 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:04.755102 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:04.755103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:04.755150 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:04.755720 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:20:04.755725 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:20:04.757817 (+ 2092us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.18.62:36865: Server oryx/127.3.18.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":90,"threads_started":1}
W20260928 20:20:04.758011 12297 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:20:04.752873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:41291 (local address 127.3.18.62:36865)
0928 20:20:04.752951 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:04.752953 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:04.753023 (+ 70us) server_negotiation.cc:407] Connection header received
0928 20:20:04.753086 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:04.753087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:04.753100 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:04.753142 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:04.753488 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:04.754028 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:04.754839 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:04.755028 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:04.755086 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:04.757910 (+ 2824us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.18.3:41291: BlockingRecv error: recv got EOF from 127.3.18.3:41291 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 28 20:20:05 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.1@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:20:05 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.1@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:20:05.500654 12300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:05.495309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36865 (local address 127.3.18.1:36491)
0928 20:20:05.495476 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:05.495489 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 20:20:05.495557 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:20:05.495801 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:20:05.495802 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:20:05.495986 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:20:05.496037 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:05.496039 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:05.496716 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:05.496717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:05.497337 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:05.497340 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:05.497831 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:05.497833 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:05.497899 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:05.498442 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:20:05.498447 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:20:05.500552 (+ 2105us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.18.62:36865: Server oryx/127.3.18.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":113,"threads_started":1}
W20260928 20:20:05.500787 12301 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:20:05.495428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:36491 (local address 127.3.18.62:36865)
0928 20:20:05.495567 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:05.495569 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:05.495578 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:20:05.495606 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:05.495607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:05.495619 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:05.495721 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:05.496109 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:05.496607 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:05.497442 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:05.497701 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:05.497817 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:05.500695 (+ 2878us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.18.1:36491: BlockingRecv error: recv got EOF from 127.3.18.1:36491 (error 108)
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":79,"threads_started":1}
Sep 28 20:20:05 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.2@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:20:05 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.2@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:20:05.629000 12302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:05.623374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36865 (local address 127.3.18.2:57859)
0928 20:20:05.623552 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:05.623568 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 20:20:05.623674 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:20:05.623980 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:20:05.623981 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:20:05.624161 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:20:05.624213 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:05.624215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:05.625013 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:05.625014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:05.625557 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:05.625559 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:05.625992 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:05.625994 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:05.626069 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:05.626689 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:20:05.626701 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:20:05.628910 (+ 2209us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.18.62:36865: Server oryx/127.3.18.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":74,"threads_started":1}
W20260928 20:20:05.629112 12301 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:20:05.623536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:57859 (local address 127.3.18.62:36865)
0928 20:20:05.623592 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:05.623595 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:05.623664 (+ 69us) server_negotiation.cc:407] Connection header received
0928 20:20:05.623790 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:05.623791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:05.623808 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:05.623889 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:05.624333 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:05.624893 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:05.625660 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:05.625875 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:05.625981 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:05.629039 (+ 3058us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.18.2:57859: BlockingRecv error: recv got EOF from 127.3.18.2:57859 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 28 20:20:05 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.3@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:20:05 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.3@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:20:05.763880 12303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:05.758437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36865 (local address 127.3.18.3:49933)
0928 20:20:05.758625 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:05.758643 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 20:20:05.758699 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:20:05.758867 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:20:05.758868 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:20:05.759093 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:20:05.759146 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:05.759148 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:05.760134 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:05.760135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:05.760745 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:05.760747 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:05.761193 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:05.761194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:05.761246 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:05.761740 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:20:05.761745 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:20:05.763755 (+ 2010us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.18.62:36865: Server oryx/127.3.18.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":132,"threads_started":1}
W20260928 20:20:05.764015 12301 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:20:05.758614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:49933 (local address 127.3.18.62:36865)
0928 20:20:05.758693 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:05.758695 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:05.758704 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:20:05.758758 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:05.758759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:05.758779 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:05.758844 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:05.759287 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:05.760015 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:05.760849 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:05.761044 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:05.761153 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:05.763910 (+ 2757us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.18.3:49933: BlockingRecv error: recv got EOF from 127.3.18.3:49933 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 28 20:20:06 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.1@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:20:06 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.1@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:20:06.506718 12304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:06.501359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36865 (local address 127.3.18.1:39425)
0928 20:20:06.501566 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:06.501578 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:20:06.501650 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:20:06.501806 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:20:06.501806 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:20:06.501987 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:20:06.502035 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:06.502037 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:06.502830 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:06.502831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:06.503400 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:06.503402 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:06.503865 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:06.503866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:06.503917 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:06.504451 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:20:06.504456 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:20:06.506569 (+ 2113us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.18.62:36865: Server oryx/127.3.18.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":123,"threads_started":1}
W20260928 20:20:06.506731 12305 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:20:06.501488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:39425 (local address 127.3.18.62:36865)
0928 20:20:06.501667 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:06.501669 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:06.501695 (+ 26us) server_negotiation.cc:407] Connection header received
0928 20:20:06.501719 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:06.501720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:06.501731 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:06.501783 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:06.502155 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:06.502726 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:06.503486 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:06.503727 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:06.503848 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:06.506665 (+ 2817us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.18.1:39425: BlockingRecv error: recv got EOF from 127.3.18.1:39425 (error 108)
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":57,"threads_started":1}
Sep 28 20:20:06 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.2@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:20:06 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.2@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:20:06.634729 12306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:06.629545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36865 (local address 127.3.18.2:35213)
0928 20:20:06.629755 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:06.629767 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:20:06.629829 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:20:06.630003 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:20:06.630004 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:20:06.630179 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:20:06.630229 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:06.630231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:06.630912 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:06.630913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:06.631461 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:06.631463 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:06.631881 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:06.631882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:06.631931 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:06.632501 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:20:06.632506 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:20:06.634578 (+ 2072us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.18.62:36865: Server oryx/127.3.18.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":107,"threads_started":1}
W20260928 20:20:06.634734 12305 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:20:06.629771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:35213 (local address 127.3.18.62:36865)
0928 20:20:06.629824 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:06.629827 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:06.629835 (+ 8us) server_negotiation.cc:407] Connection header received
0928 20:20:06.629878 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:06.629879 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:06.629890 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:06.629943 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:06.630306 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:06.630830 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:06.631559 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:06.631799 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:06.631863 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:06.634681 (+ 2818us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.18.2:35213: BlockingRecv error: recv got EOF from 127.3.18.2:35213 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 28 20:20:06 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.3@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:20:06 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.3@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:20:06.769338 12307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:06.764445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36865 (local address 127.3.18.3:47109)
0928 20:20:06.764581 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:06.764593 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:20:06.764651 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:20:06.764838 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:20:06.764839 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:20:06.765024 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:20:06.765075 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:06.765076 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:06.765859 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:06.765861 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:06.766420 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:06.766422 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:06.766806 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:06.766808 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:06.766854 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:06.767317 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:20:06.767322 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:20:06.769276 (+ 1954us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.18.62:36865: Server oryx/127.3.18.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":84,"threads_started":1}
W20260928 20:20:06.769511 12305 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:20:06.764541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:47109 (local address 127.3.18.62:36865)
0928 20:20:06.764674 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:06.764676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:06.764708 (+ 32us) server_negotiation.cc:407] Connection header received
0928 20:20:06.764744 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:06.764745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:06.764757 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:06.764828 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:06.765178 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:06.765720 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:06.766515 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:06.766725 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:06.766786 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:06.769422 (+ 2636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.18.3:47109: BlockingRecv error: recv got EOF from 127.3.18.3:47109 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Sep 28 20:20:07 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.1@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:20:07 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.1@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:20:07.517969 12308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:07.507338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36865 (local address 127.3.18.1:49351)
0928 20:20:07.507487 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:07.507498 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:20:07.507548 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:20:07.507868 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:20:07.507869 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:20:07.508045 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:20:07.508095 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:07.508097 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:07.508874 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:07.508875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:07.509394 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:07.509396 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:07.509759 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:07.509759 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:07.509802 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:07.515641 (+ 5839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:20:07.515649 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:20:07.517872 (+ 2223us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.18.62:36865: Server oryx/127.3.18.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":88,"threads_started":1}
W20260928 20:20:07.518040 12309 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:20:07.507406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:49351 (local address 127.3.18.62:36865)
0928 20:20:07.507621 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:07.507623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:07.507631 (+ 8us) server_negotiation.cc:407] Connection header received
0928 20:20:07.507655 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:07.507655 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:07.507671 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:07.507740 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:07.508204 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:07.508790 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:07.509467 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:07.509689 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:07.509744 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:07.517967 (+ 8223us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.18.1:49351: BlockingRecv error: recv got EOF from 127.3.18.1:49351 (error 108)
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":85,"threads_started":1}
Sep 28 20:20:07 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.2@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:20:07 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.2@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:20:07.640579 12310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:07.635343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36865 (local address 127.3.18.2:49727)
0928 20:20:07.635499 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:07.635510 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:20:07.635618 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:20:07.635855 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:20:07.635855 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:20:07.636025 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:20:07.636075 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:07.636078 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:07.637070 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:07.637071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:07.637625 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:07.637628 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:07.638023 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:07.638023 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:07.638062 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:07.638558 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:20:07.638563 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:20:07.640500 (+ 1937us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.18.62:36865: Server oryx/127.3.18.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":92,"threads_started":1}
W20260928 20:20:07.640821 12309 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:20:07.635453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:49727 (local address 127.3.18.62:36865)
0928 20:20:07.635507 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:07.635509 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:07.635600 (+ 91us) server_negotiation.cc:407] Connection header received
0928 20:20:07.635710 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:07.635711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:07.635722 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:07.635847 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:07.636164 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:07.636993 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:07.637757 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:07.637951 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:07.638017 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:07.640749 (+ 2732us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.18.2:49727: BlockingRecv error: recv got EOF from 127.3.18.2:49727 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 28 20:20:07 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.3@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:20:07 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.3@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:20:07.775434 12311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:07.769868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36865 (local address 127.3.18.3:45009)
0928 20:20:07.770099 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:07.770114 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 20:20:07.770248 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:20:07.770442 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:20:07.770443 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:20:07.770634 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:20:07.770724 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:07.770726 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:07.771429 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:07.771431 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:07.772002 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:07.772004 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:07.772456 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:07.772458 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:07.772526 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:07.773093 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:20:07.773098 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:20:07.775321 (+ 2223us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.18.62:36865: Server oryx/127.3.18.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":101,"threads_started":1}
W20260928 20:20:07.775501 12309 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:20:07.769984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:45009 (local address 127.3.18.62:36865)
0928 20:20:07.770103 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:07.770105 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:07.770220 (+ 115us) server_negotiation.cc:407] Connection header received
0928 20:20:07.770345 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:07.770346 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:07.770361 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:07.770419 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:07.770831 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:07.771341 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:07.772094 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:07.772341 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:07.772454 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:07.775426 (+ 2972us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.18.3:45009: BlockingRecv error: recv got EOF from 127.3.18.3:45009 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 28 20:20:08 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.1@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:20:08 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.1@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:20:08.523452 12312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:08.518626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36865 (local address 127.3.18.1:39661)
0928 20:20:08.518779 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:08.518794 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 20:20:08.518846 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:20:08.519016 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:20:08.519017 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:20:08.519187 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:20:08.519236 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:08.519238 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:08.520035 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:08.520037 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:08.520578 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:08.520580 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:08.520966 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:08.520967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:08.521009 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:08.521448 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:20:08.521452 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:20:08.523362 (+ 1910us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.18.62:36865: Server oryx/127.3.18.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":93,"threads_started":1}
W20260928 20:20:08.523540 12313 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:20:08.518687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:39661 (local address 127.3.18.62:36865)
0928 20:20:08.518841 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:08.518843 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:08.518854 (+ 11us) server_negotiation.cc:407] Connection header received
0928 20:20:08.518906 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:08.518906 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:08.518917 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:08.518984 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:08.519347 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:08.519928 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:08.520692 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:08.520891 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:08.520950 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:08.523465 (+ 2515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.18.1:39661: BlockingRecv error: recv got EOF from 127.3.18.1:39661 (error 108)
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":98,"threads_started":1}
Sep 28 20:20:08 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.2@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:20:08 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.2@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:20:08.646193 12314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:08.641170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36865 (local address 127.3.18.2:39189)
0928 20:20:08.641321 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:08.641333 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:20:08.641409 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:20:08.641557 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:20:08.641557 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:20:08.641723 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:20:08.641773 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:08.641774 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:08.642416 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:08.642417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:08.642949 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:08.642951 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:08.643338 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:08.643340 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:08.643393 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:08.644099 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:20:08.644105 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:20:08.646084 (+ 1979us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.18.62:36865: Server oryx/127.3.18.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":73,"threads_started":1}
W20260928 20:20:08.646286 12313 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:20:08.641267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:39189 (local address 127.3.18.62:36865)
0928 20:20:08.641352 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:08.641354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:08.641392 (+ 38us) server_negotiation.cc:407] Connection header received
0928 20:20:08.641470 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:08.641471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:08.641483 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:08.641536 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:08.641836 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:08.642337 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:08.643032 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:08.643237 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:08.643295 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:08.646208 (+ 2913us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.18.2:39189: BlockingRecv error: recv got EOF from 127.3.18.2:39189 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 28 20:20:08 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.3@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:20:08 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.3@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:20:08.781592 12315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:08.776079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36865 (local address 127.3.18.3:39687)
0928 20:20:08.776250 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:08.776262 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:20:08.776328 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:20:08.776492 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:20:08.776493 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:20:08.776746 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:20:08.776810 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:08.776812 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:08.777553 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:08.777554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:08.778072 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:08.778074 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:08.778445 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:08.778446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:08.778527 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:08.779026 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:20:08.779030 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:20:08.781450 (+ 2420us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.18.62:36865: Server oryx/127.3.18.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":161,"threads_started":1}
W20260928 20:20:08.781628 12313 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:20:08.776263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:39687 (local address 127.3.18.62:36865)
0928 20:20:08.776329 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:08.776331 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:08.776341 (+ 10us) server_negotiation.cc:407] Connection header received
0928 20:20:08.776400 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:08.776405 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:08.776420 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:08.776492 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:08.776926 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:08.777465 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:08.778166 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:08.778358 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:08.778433 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:08.781567 (+ 3134us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.18.3:39687: BlockingRecv error: recv got EOF from 127.3.18.3:39687 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 28 20:20:09 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.1@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:20:09 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.1@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:20:09.528671 12316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:09.523921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36865 (local address 127.3.18.1:40711)
0928 20:20:09.524075 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:09.524088 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 20:20:09.524185 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:20:09.524388 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:20:09.524389 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:20:09.524555 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:20:09.524601 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:09.524602 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:09.525284 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:09.525284 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:09.525813 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:09.525814 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:09.526202 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:09.526203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:09.526258 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:09.526755 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:20:09.526760 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:20:09.528596 (+ 1836us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.18.62:36865: Server oryx/127.3.18.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":57,"threads_started":1}
W20260928 20:20:09.528788 12317 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:20:09.524132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:40711 (local address 127.3.18.62:36865)
0928 20:20:09.524253 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:09.524256 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:09.524266 (+ 10us) server_negotiation.cc:407] Connection header received
0928 20:20:09.524298 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:09.524298 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:09.524308 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:09.524367 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:09.524668 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:09.525212 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:09.525901 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:09.526098 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:09.526154 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:09.528708 (+ 2554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.18.1:40711: BlockingRecv error: recv got EOF from 127.3.18.1:40711 (error 108)
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":105,"threads_started":1}
Sep 28 20:20:09 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.2@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:20:09 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.2@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:20:09.651454 12318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:09.646698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36865 (local address 127.3.18.2:40931)
0928 20:20:09.646871 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:09.646889 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 20:20:09.646945 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:20:09.647094 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:20:09.647095 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:20:09.647274 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:20:09.647323 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:09.647324 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:09.648222 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:09.648222 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:09.648746 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:09.648748 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:09.649091 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:09.649092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:09.649136 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:09.649605 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:20:09.649610 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:20:09.651395 (+ 1785us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.18.62:36865: Server oryx/127.3.18.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":116,"threads_started":1}
W20260928 20:20:09.651584 12317 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:20:09.646794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:40931 (local address 127.3.18.62:36865)
0928 20:20:09.646849 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:09.646850 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:09.646932 (+ 82us) server_negotiation.cc:407] Connection header received
0928 20:20:09.647007 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:09.647008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:09.647019 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:09.647073 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:09.647394 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:09.648143 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:09.648822 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:09.649019 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:09.649076 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:09.651497 (+ 2421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.18.2:40931: BlockingRecv error: recv got EOF from 127.3.18.2:40931 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 28 20:20:09 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.3@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:20:09 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.3@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:20:09.787523 12319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:09.782475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36865 (local address 127.3.18.3:33017)
0928 20:20:09.782647 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:09.782658 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:20:09.782736 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:20:09.782895 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:20:09.782895 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:20:09.783085 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:20:09.783134 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:09.783136 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:09.783881 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:09.783882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:09.784393 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:09.784394 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:09.784756 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:09.784757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:09.784817 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:09.785364 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:20:09.785368 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:20:09.787465 (+ 2097us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.18.62:36865: Server oryx/127.3.18.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":105,"threads_started":1}
W20260928 20:20:09.787604 12317 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:20:09.782578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:33017 (local address 127.3.18.62:36865)
0928 20:20:09.782656 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:09.782658 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:09.782740 (+ 82us) server_negotiation.cc:407] Connection header received
0928 20:20:09.782786 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:09.782787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:09.782801 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:09.782864 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:09.783251 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:09.783750 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:09.784472 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:09.784653 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:09.784711 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:09.787555 (+ 2844us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.18.3:33017: BlockingRecv error: recv got EOF from 127.3.18.3:33017 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 28 20:20:10 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.1@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:20:10 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.1@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:20:10.535280 12320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:10.529437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36865 (local address 127.3.18.1:54417)
0928 20:20:10.529639 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:10.529654 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 20:20:10.529737 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:20:10.530016 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:20:10.530017 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:20:10.530216 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:20:10.530267 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:10.530269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:10.530950 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:10.530951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:10.531561 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:10.531563 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:10.532080 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:10.532081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:10.532140 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:10.532919 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:20:10.532924 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:20:10.535215 (+ 2291us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.18.62:36865: Server oryx/127.3.18.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":86,"threads_started":1}
W20260928 20:20:10.535434 12321 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:20:10.529603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:54417 (local address 127.3.18.62:36865)
0928 20:20:10.529833 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:10.529835 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:10.529844 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:20:10.529876 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:10.529877 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:10.529888 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:10.529954 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:10.530354 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:10.530868 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:10.531700 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:10.532002 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:10.532059 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:10.535333 (+ 3274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.18.1:54417: BlockingRecv error: recv got EOF from 127.3.18.1:54417 (error 108)
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":93,"threads_started":1}
Sep 28 20:20:10 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.2@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:20:10 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.2@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:20:10.657629 12322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:10.652133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36865 (local address 127.3.18.2:59367)
0928 20:20:10.652352 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:10.652365 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 20:20:10.652481 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:20:10.652647 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:20:10.652648 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:20:10.652872 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:20:10.652922 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:10.652924 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:10.653655 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:10.653656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:10.654259 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:10.654263 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:10.654799 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:10.654801 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:10.654881 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:10.655483 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:20:10.655494 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:20:10.657568 (+ 2074us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.18.62:36865: Server oryx/127.3.18.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":120,"threads_started":1}
W20260928 20:20:10.657786 12321 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:20:10.652286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:59367 (local address 127.3.18.62:36865)
0928 20:20:10.652345 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:10.652347 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:10.652459 (+ 112us) server_negotiation.cc:407] Connection header received
0928 20:20:10.652560 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:10.652561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:10.652574 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:10.652629 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:10.653037 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:10.653558 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:10.654368 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:10.654617 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:10.654693 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:10.657695 (+ 3002us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.18.2:59367: BlockingRecv error: recv got EOF from 127.3.18.2:59367 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 28 20:20:10 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.3@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:20:10 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.3@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:20:10.792986 12323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:10.788155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36865 (local address 127.3.18.3:44735)
0928 20:20:10.788330 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:10.788346 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 20:20:10.788469 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:20:10.788628 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:20:10.788629 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:20:10.788823 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:20:10.788876 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:10.788879 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:10.789518 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:10.789518 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:10.790115 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:10.790116 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:10.790477 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:10.790478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:10.790523 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:10.790988 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:20:10.790993 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:20:10.792922 (+ 1929us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.18.62:36865: Server oryx/127.3.18.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":88,"threads_started":1}
W20260928 20:20:10.793195 12321 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:20:10.788232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:44735 (local address 127.3.18.62:36865)
0928 20:20:10.788285 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:10.788287 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:10.788451 (+ 164us) server_negotiation.cc:407] Connection header received
0928 20:20:10.788545 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:10.788546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:10.788558 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:10.788596 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:10.788955 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:10.789447 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:10.790209 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:10.790402 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:10.790464 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:10.793123 (+ 2659us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.18.3:44735: BlockingRecv error: recv got EOF from 127.3.18.3:44735 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 28 20:20:11 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.1@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:20:11 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.1@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:20:11.541334 12324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:11.536017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36865 (local address 127.3.18.1:59971)
0928 20:20:11.536196 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:11.536210 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 20:20:11.536327 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:20:11.536495 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:20:11.536495 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:20:11.536677 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:20:11.536726 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:11.536728 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:11.537698 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:11.537700 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:11.538264 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:11.538266 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:11.538632 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:11.538633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:11.538681 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:11.539177 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:20:11.539182 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:20:11.541235 (+ 2053us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.18.62:36865: Server oryx/127.3.18.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":113,"threads_started":1}
W20260928 20:20:11.541409 12325 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:20:11.536140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:59971 (local address 127.3.18.62:36865)
0928 20:20:11.536292 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:11.536299 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:11.536312 (+ 13us) server_negotiation.cc:407] Connection header received
0928 20:20:11.536384 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:11.536385 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:11.536398 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:11.536463 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:11.536832 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:11.537606 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:11.538354 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:11.538554 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:11.538613 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:11.541333 (+ 2720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.18.1:59971: BlockingRecv error: recv got EOF from 127.3.18.1:59971 (error 108)
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":72,"threads_started":1}
Sep 28 20:20:11 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.2@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:20:11 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.2@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:20:11.664314 12326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:11.658299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36865 (local address 127.3.18.2:49031)
0928 20:20:11.658476 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:11.658560 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0928 20:20:11.658650 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:20:11.658850 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:20:11.658851 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:20:11.659097 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:20:11.659146 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:11.659148 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:11.660391 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:11.660392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:11.661097 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:11.661100 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:11.661496 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:11.661497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:11.661546 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:11.662076 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:20:11.662081 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:20:11.664210 (+ 2129us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.18.62:36865: Server oryx/127.3.18.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":99,"threads_started":1}
W20260928 20:20:11.664373 12325 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:20:11.658444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:49031 (local address 127.3.18.62:36865)
0928 20:20:11.658517 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:11.658519 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:11.658698 (+ 179us) server_negotiation.cc:407] Connection header received
0928 20:20:11.658733 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:11.658734 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:11.658747 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:11.658808 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:11.659271 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:11.660286 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:11.661202 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:11.661410 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:11.661473 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:11.664303 (+ 2830us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.18.2:49031: BlockingRecv error: recv got EOF from 127.3.18.2:49031 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 28 20:20:11 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.3@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:20:11 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.3@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:20:11.798694 12327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:11.793560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36865 (local address 127.3.18.3:60599)
0928 20:20:11.793677 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:11.793689 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:20:11.793816 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:20:11.793987 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:20:11.793988 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:20:11.794175 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:20:11.794239 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:11.794242 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:11.794940 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:11.794950 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:11.795524 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:11.795526 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:11.796049 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:11.796050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:11.796111 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:11.796597 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:20:11.796602 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:20:11.798605 (+ 2003us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.18.62:36865: Server oryx/127.3.18.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":70,"threads_started":1}
W20260928 20:20:11.798808 12325 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:20:11.793713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:60599 (local address 127.3.18.62:36865)
0928 20:20:11.793788 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:11.793791 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:11.793801 (+ 10us) server_negotiation.cc:407] Connection header received
0928 20:20:11.793879 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:11.793880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:11.793895 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:11.793975 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:11.794336 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:11.794843 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:11.795642 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:11.795943 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:11.796050 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:11.798727 (+ 2677us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.18.3:60599: BlockingRecv error: recv got EOF from 127.3.18.3:60599 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 28 20:20:12 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.1@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:20:12 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.1@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:20:12.550297 12328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:12.541885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36865 (local address 127.3.18.1:47297)
0928 20:20:12.542048 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:12.542059 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:20:12.542110 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:20:12.542292 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:20:12.542293 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:20:12.542453 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:20:12.542511 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:12.542513 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:12.543245 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:12.543246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:12.544097 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:12.544100 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:12.544470 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:12.544471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:12.544516 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:12.548247 (+ 3731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:20:12.548254 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:20:12.550237 (+ 1983us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.18.62:36865: Server oryx/127.3.18.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":96,"threads_started":1}
W20260928 20:20:12.550521 12329 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:20:12.542010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:47297 (local address 127.3.18.62:36865)
0928 20:20:12.542143 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:12.542145 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:12.542154 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:20:12.542178 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:12.542178 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:12.542189 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:12.542267 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:12.542587 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:12.543130 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:12.544177 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:12.544399 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:12.544454 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:12.550418 (+ 5964us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.18.1:47297: BlockingRecv error: recv got EOF from 127.3.18.1:47297 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":74,"threads_started":1}
Sep 28 20:20:12 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.2@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:20:12 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.2@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:20:12.669903 12330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:12.664956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36865 (local address 127.3.18.2:49497)
0928 20:20:12.665150 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:12.665162 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:20:12.665224 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:20:12.665410 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:20:12.665411 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:20:12.665576 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:20:12.665673 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:12.665675 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:12.666384 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:12.666385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:12.666959 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:12.666961 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:12.667387 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:12.667387 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:12.667433 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:12.668075 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:20:12.668085 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:20:12.669842 (+ 1757us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.18.62:36865: Server oryx/127.3.18.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":101,"threads_started":1}
W20260928 20:20:12.670070 12329 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:20:12.665107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:49497 (local address 127.3.18.62:36865)
0928 20:20:12.665189 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:12.665191 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:12.665205 (+ 14us) server_negotiation.cc:407] Connection header received
0928 20:20:12.665275 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:12.665276 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:12.665288 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:12.665352 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:12.665776 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:12.666305 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:12.667048 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:12.667277 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:12.667356 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:12.669983 (+ 2627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.18.2:49497: BlockingRecv error: recv got EOF from 127.3.18.2:49497 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 28 20:20:12 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.3@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:20:12 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.3@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:20:12.804280 12331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:12.799306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36865 (local address 127.3.18.3:47657)
0928 20:20:12.799426 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:12.799439 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 20:20:12.799525 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:20:12.799683 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:20:12.799684 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:20:12.799875 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:20:12.799927 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:12.799929 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:12.800618 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:12.800619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:12.801174 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:12.801176 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:12.801527 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:12.801528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:12.801578 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:12.802092 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:20:12.802097 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:20:12.804220 (+ 2123us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.18.62:36865: Server oryx/127.3.18.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":56,"threads_started":1}
W20260928 20:20:12.804438 12329 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:20:12.799398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:47657 (local address 127.3.18.62:36865)
0928 20:20:12.799456 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:12.799458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:12.799500 (+ 42us) server_negotiation.cc:407] Connection header received
0928 20:20:12.799589 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:12.799590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:12.799602 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:12.799668 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:12.800002 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:12.800523 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:12.801251 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:12.801435 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:12.801491 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:12.804338 (+ 2847us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.18.3:47657: BlockingRecv error: recv got EOF from 127.3.18.3:47657 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 28 20:20:13 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.1@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:20:13 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.1@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:20:13.555797 12332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:13.550847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36865 (local address 127.3.18.1:43443)
0928 20:20:13.551001 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:13.551012 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:20:13.551068 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:20:13.551257 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:20:13.551258 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:20:13.551431 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:20:13.551480 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:13.551481 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:13.552315 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:13.552316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:13.552863 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:13.552865 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:13.553225 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:13.553226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:13.553272 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:13.553750 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:20:13.553754 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:20:13.555686 (+ 1932us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.18.62:36865: Server oryx/127.3.18.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":89,"threads_started":1}
W20260928 20:20:13.555997 12333 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:20:13.550955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:43443 (local address 127.3.18.62:36865)
0928 20:20:13.551094 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:13.551096 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:13.551104 (+ 8us) server_negotiation.cc:407] Connection header received
0928 20:20:13.551144 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:13.551145 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:13.551158 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:13.551234 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:13.551570 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:13.552221 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:13.552957 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:13.553153 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:13.553210 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:13.555919 (+ 2709us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.18.1:43443: BlockingRecv error: recv got EOF from 127.3.18.1:43443 (error 108)
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":83,"threads_started":1}
Sep 28 20:20:13 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.2@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:20:13 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.2@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:20:13.675333 12334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:13.670433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36865 (local address 127.3.18.2:51645)
0928 20:20:13.670580 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:13.670596 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 20:20:13.670649 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:20:13.670795 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:20:13.670796 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:20:13.670970 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:20:13.671018 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:13.671026 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:13.671827 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:13.671828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:13.672380 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:13.672383 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:13.672841 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:13.672842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:13.672891 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:13.673513 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:20:13.673517 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:20:13.675273 (+ 1756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.18.62:36865: Server oryx/127.3.18.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":88,"threads_started":1}
W20260928 20:20:13.675506 12333 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:20:13.670565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:51645 (local address 127.3.18.62:36865)
0928 20:20:13.670613 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:13.670615 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:13.670645 (+ 30us) server_negotiation.cc:407] Connection header received
0928 20:20:13.670718 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:13.670719 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:13.670730 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:13.670774 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:13.671130 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:13.671653 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:13.672485 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:13.672742 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:13.672810 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:13.675435 (+ 2625us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.18.2:51645: BlockingRecv error: recv got EOF from 127.3.18.2:51645 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 28 20:20:13 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.3@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:20:13 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.3@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:20:13.809636 12335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:13.804825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36865 (local address 127.3.18.3:33973)
0928 20:20:13.805034 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:13.805045 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:20:13.805120 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:20:13.805278 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:20:13.805279 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:20:13.805482 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:20:13.805533 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:13.805535 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:13.806231 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:13.806233 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:13.806770 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:13.806771 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:13.807162 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:13.807163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:13.807224 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:13.807708 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:20:13.807712 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:20:13.809562 (+ 1850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.18.62:36865: Server oryx/127.3.18.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":117,"threads_started":1}
W20260928 20:20:13.809845 12333 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:20:13.804926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:33973 (local address 127.3.18.62:36865)
0928 20:20:13.805006 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:13.805009 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:13.805145 (+ 136us) server_negotiation.cc:407] Connection header received
0928 20:20:13.805175 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:13.805176 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:13.805192 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:13.805270 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:13.805620 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:13.806144 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:13.806869 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:13.807057 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:13.807152 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:13.809755 (+ 2603us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.18.3:33973: BlockingRecv error: recv got EOF from 127.3.18.3:33973 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 28 20:20:14 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.1@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:20:14 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.1@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:20:14.561296 12336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:14.556391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36865 (local address 127.3.18.1:53349)
0928 20:20:14.556531 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:14.556544 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 20:20:14.556612 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:20:14.556767 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:20:14.556768 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:20:14.556937 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:20:14.556986 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:14.556987 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:14.557706 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:14.557707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:14.558261 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:14.558263 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:14.558669 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:14.558670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:14.558731 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:14.559248 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:20:14.559253 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:20:14.561235 (+ 1982us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.18.62:36865: Server oryx/127.3.18.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":74,"threads_started":1}
W20260928 20:20:14.561507 12337 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:20:14.556463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:53349 (local address 127.3.18.62:36865)
0928 20:20:14.556597 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:14.556599 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:14.556608 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:20:14.556668 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:14.556668 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:14.556679 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:14.556735 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:14.557102 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:14.557621 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:14.558348 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:14.558566 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:14.558636 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:14.561420 (+ 2784us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.18.1:53349: BlockingRecv error: recv got EOF from 127.3.18.1:53349 (error 108)
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":70,"threads_started":1}
Sep 28 20:20:14 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.2@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:20:14 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.2@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:20:14.681385 12338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:14.675975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36865 (local address 127.3.18.2:60471)
0928 20:20:14.676258 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:14.676270 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:20:14.676362 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:20:14.676548 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:20:14.676549 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:20:14.676722 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:20:14.676768 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:14.676770 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:14.677557 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:14.677559 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:14.678326 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:14.678330 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:14.678695 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:14.678696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:14.678754 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:14.679331 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:20:14.679336 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:20:14.681312 (+ 1976us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.18.62:36865: Server oryx/127.3.18.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":159,"threads_started":1}
W20260928 20:20:14.681490 12337 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:20:14.676183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:60471 (local address 127.3.18.62:36865)
0928 20:20:14.676243 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:14.676245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:14.676367 (+ 122us) server_negotiation.cc:407] Connection header received
0928 20:20:14.676435 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:14.676436 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:14.676449 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:14.676514 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:14.676839 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:14.677401 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:14.678403 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:14.678609 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:14.678698 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:14.681417 (+ 2719us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.18.2:60471: BlockingRecv error: recv got EOF from 127.3.18.2:60471 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 28 20:20:14 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.3@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:20:14 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.3@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:20:14.816008 12339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:14.810343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36865 (local address 127.3.18.3:32927)
0928 20:20:14.810540 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:14.810553 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 20:20:14.810648 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:20:14.810860 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:20:14.810861 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:20:14.811078 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:20:14.811162 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:14.811166 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:14.812156 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:14.812157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:14.812694 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:14.812696 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:14.813056 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:14.813057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:14.813099 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:14.813603 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:20:14.813608 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:20:14.815938 (+ 2330us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.18.62:36865: Server oryx/127.3.18.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":79,"threads_started":1}
W20260928 20:20:14.816144 12337 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:20:14.810465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:32927 (local address 127.3.18.62:36865)
0928 20:20:14.810544 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:14.810546 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:14.810690 (+ 144us) server_negotiation.cc:407] Connection header received
0928 20:20:14.810733 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:14.810735 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:14.810751 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:14.810835 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:14.811290 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:14.812032 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:14.812796 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:14.812985 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:14.813041 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:14.816066 (+ 3025us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.18.3:32927: BlockingRecv error: recv got EOF from 127.3.18.3:32927 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 28 20:20:15 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.1@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:20:15 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.1@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:20:15.567214 12340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:15.561901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36865 (local address 127.3.18.1:43265)
0928 20:20:15.562049 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:15.562060 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:20:15.562114 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:20:15.562420 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:20:15.562421 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:20:15.562586 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:20:15.562635 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:15.562637 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:15.563408 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:15.563409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:15.564087 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:15.564090 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:15.564534 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:15.564536 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:15.564606 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:15.565134 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:20:15.565139 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:20:15.567141 (+ 2002us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.18.62:36865: Server oryx/127.3.18.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":83,"threads_started":1}
W20260928 20:20:15.567435 12341 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:20:15.562032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:43265 (local address 127.3.18.62:36865)
0928 20:20:15.562261 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:15.562264 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:15.562274 (+ 10us) server_negotiation.cc:407] Connection header received
0928 20:20:15.562301 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:15.562302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:15.562313 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:15.562391 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:15.562795 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:15.563313 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:15.564190 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:15.564447 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:15.564533 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:15.567302 (+ 2769us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.18.1:43265: BlockingRecv error: recv got EOF from 127.3.18.1:43265 (error 108)
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":167,"threads_started":1}
Sep 28 20:20:15 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.2@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:20:15 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.2@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:20:15.687749 12342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:15.681990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36865 (local address 127.3.18.2:33431)
0928 20:20:15.682169 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:15.682189 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 20:20:15.682247 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:20:15.682447 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:20:15.682448 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:20:15.682614 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:20:15.682662 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:15.682663 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:15.683668 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:15.683668 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:15.684356 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:15.684359 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:15.684812 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:15.684813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:15.684861 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:15.685414 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:20:15.685419 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:20:15.687639 (+ 2220us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.18.62:36865: Server oryx/127.3.18.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":115,"threads_started":1}
W20260928 20:20:15.687888 12341 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:20:15.682137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:33431 (local address 127.3.18.62:36865)
0928 20:20:15.682216 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:15.682217 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:15.682232 (+ 15us) server_negotiation.cc:407] Connection header received
0928 20:20:15.682320 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:15.682321 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:15.682332 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:15.682395 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:15.682834 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:15.683551 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:15.684473 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:15.684699 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:15.684772 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:15.687804 (+ 3032us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.18.2:33431: BlockingRecv error: recv got EOF from 127.3.18.2:33431 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 28 20:20:15 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.3@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:20:15 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.3@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:20:15.821938 12343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:15.816624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36865 (local address 127.3.18.3:53053)
0928 20:20:15.816777 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:15.816790 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 20:20:15.816863 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:20:15.817088 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:20:15.817090 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:20:15.817254 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:20:15.817329 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:15.817332 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:15.818201 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:15.818202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:15.818776 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:15.818778 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:15.819156 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:15.819157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:15.819207 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:15.819938 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:20:15.819942 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:20:15.821867 (+ 1925us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.18.62:36865: Server oryx/127.3.18.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":83,"threads_started":1}
W20260928 20:20:15.822108 12341 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:20:15.816677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:53053 (local address 127.3.18.62:36865)
0928 20:20:15.816793 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:15.816795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:15.816870 (+ 75us) server_negotiation.cc:407] Connection header received
0928 20:20:15.816963 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:15.816964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:15.816979 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:15.817032 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:15.817451 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:15.818108 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:15.818870 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:15.819056 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:15.819123 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:15.822023 (+ 2900us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.18.3:53053: BlockingRecv error: recv got EOF from 127.3.18.3:53053 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 28 20:20:16 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.1@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:20:16 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.1@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:20:16.572885 12344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:16.567951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36865 (local address 127.3.18.1:58669)
0928 20:20:16.568091 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:16.568110 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 20:20:16.568159 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:20:16.568402 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:20:16.568403 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:20:16.568569 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:20:16.568618 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:16.568620 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:16.569374 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:16.569375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:16.569902 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:16.569904 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:16.570314 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:16.570315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:16.570365 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:16.570876 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:20:16.570881 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:20:16.572825 (+ 1944us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.18.62:36865: Server oryx/127.3.18.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":86,"threads_started":1}
W20260928 20:20:16.573068 12345 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:20:16.568123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:58669 (local address 127.3.18.62:36865)
0928 20:20:16.568255 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:16.568257 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:16.568275 (+ 18us) server_negotiation.cc:407] Connection header received
0928 20:20:16.568301 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:16.568302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:16.568314 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:16.568390 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:16.568720 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:16.569282 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:16.569977 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:16.570179 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:16.570232 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:16.572974 (+ 2742us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.18.1:58669: BlockingRecv error: recv got EOF from 127.3.18.1:58669 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":81,"threads_started":1}
Sep 28 20:20:16 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.2@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:20:16 dist-test-slave-b83v krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.18.2@KRBTEST.COM for oryx/127.3.18.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:20:16.693434 12346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:16.688348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36865 (local address 127.3.18.2:39151)
0928 20:20:16.688548 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:16.688559 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:20:16.688634 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:20:16.688781 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:20:16.688781 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:20:16.688946 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:20:16.688996 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:16.688997 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:16.689636 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:16.689636 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:16.690190 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:16.690192 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:16.690546 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:16.690547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:16.690587 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:16.691070 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:20:16.691075 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:20:16.693365 (+ 2290us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.18.62:36865: Server oryx/127.3.18.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":116,"threads_started":1}
W20260928 20:20:16.693553 12345 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:20:16.688406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:39151 (local address 127.3.18.62:36865)
0928 20:20:16.688477 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:16.688480 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:16.688659 (+ 179us) server_negotiation.cc:407] Connection header received
0928 20:20:16.688692 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:16.688693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:16.688705 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:16.688764 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:16.689069 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:16.689567 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:16.690285 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:16.690476 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:16.690534 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:16.693458 (+ 2924us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.18.2:39151: BlockingRecv error: recv got EOF from 127.3.18.2:39151 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 20:20:16.725749 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 11870
I20260928 20:20:16.730621 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 12022
I20260928 20:20:16.735321 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 12155
I20260928 20:20:16.739874 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 11800
2026-09-28T20:20:16Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17583 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T20:20:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:20:16Z Disabled control of system clock
I20260928 20:20:16.769069 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790626764250411-3144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790626764250411-3144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790626764250411-3144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:36239
--webserver_interface=127.3.18.62
--webserver_port=0
--builtin_ntp_servers=127.3.18.20:36781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.18.62:36239
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 20:20:16.854480 12350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:20:16.854723 12350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:20:16.854748 12350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:20:16.856207 12350 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:20:16.856256 12350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:20:16.856269 12350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:20:16.856281 12350 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:20:16.856293 12350 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:20:16.857788 12350 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:36781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790626764250411-3144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790626764250411-3144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.18.62:36239
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:36239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.18.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790626764250411-3144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:20:16.858074 12350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:20:16.858405 12350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 20:20:16.864195 12350 server_base.cc:1034] running on GCE node
W20260928 20:20:16.864220 12355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:20:16.864174 12356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:20:16.864167 12358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:20:16.864671 12350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:20:16.865023 12350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:20:16.866221 12350 hybrid_clock.cc:648] HybridClock initialized: now 1790626816866196 us; error 44 us; skew 500 ppm
I20260928 20:20:16.867656 12350 webserver.cc:460] Webserver started at http://127.3.18.62:34901/ using document root <none> and password file <none>
I20260928 20:20:16.867955 12350 fs_manager.cc:359] Metadata directory not provided
I20260928 20:20:16.868038 12350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:20:16.868209 12350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:20:16.869366 12350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790626764250411-3144-0/minicluster-data/master-0/data/instance:
uuid: "c3aa226bfd8f49b1823eb5c058e562db"
format_stamp: "Formatted at 2026-09-28 20:20:16 on dist-test-slave-b83v"
server_key: "a15eda173bb4525a24ca554392bddd2a"
server_key_iv: "712b3d534920c2f4ffce182c08a86194"
server_key_version: "encryptionkey@0"
I20260928 20:20:16.869751 12350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790626764250411-3144-0/minicluster-data/master-0/wal/instance:
uuid: "c3aa226bfd8f49b1823eb5c058e562db"
format_stamp: "Formatted at 2026-09-28 20:20:16 on dist-test-slave-b83v"
server_key: "a15eda173bb4525a24ca554392bddd2a"
server_key_iv: "712b3d534920c2f4ffce182c08a86194"
server_key_version: "encryptionkey@0"
I20260928 20:20:16.871197 12350 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:20:16.872174 12364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:20:16.872406 12350 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:20:16.872510 12350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790626764250411-3144-0/minicluster-data/master-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790626764250411-3144-0/minicluster-data/master-0/wal
uuid: "c3aa226bfd8f49b1823eb5c058e562db"
format_stamp: "Formatted at 2026-09-28 20:20:16 on dist-test-slave-b83v"
server_key: "a15eda173bb4525a24ca554392bddd2a"
server_key_iv: "712b3d534920c2f4ffce182c08a86194"
server_key_version: "encryptionkey@0"
I20260928 20:20:16.872598 12350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790626764250411-3144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790626764250411-3144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790626764250411-3144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:20:16.883477 12350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:20:16.884019 12350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:20:16.884220 12350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:20:16.888476 12350 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.62:36239
I20260928 20:20:16.888532 12416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.62:36239 every 8 connection(s)
I20260928 20:20:16.888934 12350 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
I20260928 20:20:16.889693 12417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:20:16.892385 12417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3aa226bfd8f49b1823eb5c058e562db: Bootstrap starting.
I20260928 20:20:16.893041 12417 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3aa226bfd8f49b1823eb5c058e562db: Neither blocks nor log segments found. Creating new log.
I20260928 20:20:16.893379 12417 log.cc:824] T 00000000000000000000000000000000 P c3aa226bfd8f49b1823eb5c058e562db: Log is configured to *not* fsync() on all Append() calls
I20260928 20:20:16.893949 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 12350
I20260928 20:20:16.894054 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790626764250411-3144-0/minicluster-data/master-0/wal/instance
I20260928 20:20:16.894246 12417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3aa226bfd8f49b1823eb5c058e562db: No bootstrap required, opened a new log
I20260928 20:20:16.894279 3144 external_mini_cluster.cc:1428] Setting key 8b74f03d119e78700ee07f69b897f700
I20260928 20:20:16.896123 12417 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3aa226bfd8f49b1823eb5c058e562db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3aa226bfd8f49b1823eb5c058e562db" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36239 } }
I20260928 20:20:16.896296 12417 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3aa226bfd8f49b1823eb5c058e562db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:20:16.896365 12417 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3aa226bfd8f49b1823eb5c058e562db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3aa226bfd8f49b1823eb5c058e562db, State: Initialized, Role: FOLLOWER
I20260928 20:20:16.896507 12417 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3aa226bfd8f49b1823eb5c058e562db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3aa226bfd8f49b1823eb5c058e562db" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36239 } }
I20260928 20:20:16.896587 12417 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3aa226bfd8f49b1823eb5c058e562db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:20:16.896625 12417 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3aa226bfd8f49b1823eb5c058e562db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:20:16.896674 12417 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3aa226bfd8f49b1823eb5c058e562db [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:20:16.897344 12417 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3aa226bfd8f49b1823eb5c058e562db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3aa226bfd8f49b1823eb5c058e562db" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36239 } }
I20260928 20:20:16.897454 12417 leader_election.cc:302] T 00000000000000000000000000000000 P c3aa226bfd8f49b1823eb5c058e562db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3aa226bfd8f49b1823eb5c058e562db; no voters:
I20260928 20:20:16.897689 12417 leader_election.cc:288] T 00000000000000000000000000000000 P c3aa226bfd8f49b1823eb5c058e562db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:20:16.897745 12422 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3aa226bfd8f49b1823eb5c058e562db [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:20:16.898066 12417 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3aa226bfd8f49b1823eb5c058e562db [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:20:16.898154 12422 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3aa226bfd8f49b1823eb5c058e562db [term 1 LEADER]: Becoming Leader. State: Replica: c3aa226bfd8f49b1823eb5c058e562db, State: Running, Role: LEADER
I20260928 20:20:16.898391 12422 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3aa226bfd8f49b1823eb5c058e562db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3aa226bfd8f49b1823eb5c058e562db" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36239 } }
I20260928 20:20:16.898836 12423 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3aa226bfd8f49b1823eb5c058e562db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3aa226bfd8f49b1823eb5c058e562db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3aa226bfd8f49b1823eb5c058e562db" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36239 } } }
I20260928 20:20:16.898938 12423 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3aa226bfd8f49b1823eb5c058e562db [sys.catalog]: This master's current role is: LEADER
I20260928 20:20:16.899165 12424 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3aa226bfd8f49b1823eb5c058e562db [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3aa226bfd8f49b1823eb5c058e562db. Latest consensus state: current_term: 1 leader_uuid: "c3aa226bfd8f49b1823eb5c058e562db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3aa226bfd8f49b1823eb5c058e562db" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36239 } } }
I20260928 20:20:16.899298 12424 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3aa226bfd8f49b1823eb5c058e562db [sys.catalog]: This master's current role is: LEADER
I20260928 20:20:16.899530 12431 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:20:16.900161 12431 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:20:16.902034 12431 catalog_manager.cc:1347] Generated new cluster ID: b52e8d8044024eb69daf092fb6db1c97
I20260928 20:20:16.902055 12421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:16.894942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48242 (local address 127.3.18.62:36239)
0928 20:20:16.895140 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:16.895148 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:16.895179 (+ 31us) server_negotiation.cc:407] Connection header received
0928 20:20:16.896140 (+ 961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:16.896150 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:16.896238 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:16.896426 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:16.896725 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:16.898576 (+ 1851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:16.899815 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:16.900381 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:16.900513 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:16.901302 (+ 789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:20:16.901305 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:20:16.901306 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:20:16.901541 (+ 235us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:20:16.901582 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:20:16.901598 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:20:16.901677 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 20:20:16.901752 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 20:20:16.901834 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":77,"threads_started":1}
I20260928 20:20:16.902103 12431 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:20:16.913569 12431 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:20:16.914324 12431 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:20:16.920934 12431 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3aa226bfd8f49b1823eb5c058e562db: Generated new TSK 0
I20260928 20:20:16.921141 12431 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 20:20:16.922233 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.1:0
--local_ip_for_outbound_sockets=127.3.18.1
--webserver_interface=127.3.18.1
--webserver_port=0
--tserver_master_addrs=127.3.18.62:36239
--builtin_ntp_servers=127.3.18.20:36781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 20:20:17.029948 12441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:20:17.030155 12441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:20:17.030181 12441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:20:17.031350 12441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:20:17.031392 12441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:20:17.031418 12441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.1
I20260928 20:20:17.032825 12441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:36781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.18.1
--webserver_port=0
--tserver_master_addrs=127.3.18.62:36239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12441
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.1
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:20:17.033077 12441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:20:17.033412 12441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:20:17.039273 12447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:20:17.039294 12449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:20:17.039304 12446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:20:17.039475 12441 server_base.cc:1034] running on GCE node
I20260928 20:20:17.039731 12441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:20:17.040064 12441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:20:17.041232 12441 hybrid_clock.cc:648] HybridClock initialized: now 1790626817041209 us; error 38 us; skew 500 ppm
I20260928 20:20:17.042623 12441 webserver.cc:460] Webserver started at http://127.3.18.1:40597/ using document root <none> and password file <none>
I20260928 20:20:17.042932 12441 fs_manager.cc:359] Metadata directory not provided
I20260928 20:20:17.043005 12441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:20:17.043161 12441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:20:17.044354 12441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-0/data/instance:
uuid: "c8ed292aad9b4329a9b4e563c0144597"
format_stamp: "Formatted at 2026-09-28 20:20:17 on dist-test-slave-b83v"
server_key: "850871024577f198fd4d177b2f088ebf"
server_key_iv: "77d4363b286eaee96614e5ff42e47f72"
server_key_version: "encryptionkey@0"
I20260928 20:20:17.044734 12441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-0/wal/instance:
uuid: "c8ed292aad9b4329a9b4e563c0144597"
format_stamp: "Formatted at 2026-09-28 20:20:17 on dist-test-slave-b83v"
server_key: "850871024577f198fd4d177b2f088ebf"
server_key_iv: "77d4363b286eaee96614e5ff42e47f72"
server_key_version: "encryptionkey@0"
I20260928 20:20:17.046095 12441 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 20:20:17.046870 12455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:20:17.047130 12441 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:20:17.047235 12441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-0/wal
uuid: "c8ed292aad9b4329a9b4e563c0144597"
format_stamp: "Formatted at 2026-09-28 20:20:17 on dist-test-slave-b83v"
server_key: "850871024577f198fd4d177b2f088ebf"
server_key_iv: "77d4363b286eaee96614e5ff42e47f72"
server_key_version: "encryptionkey@0"
I20260928 20:20:17.047330 12441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:20:17.072190 12441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:20:17.072474 12441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:20:17.072582 12441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:20:17.072806 12441 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:20:17.073096 12441 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:20:17.073124 12441 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:20:17.073146 12441 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:20:17.073159 12441 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:20:17.079200 12441 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.1:34317
I20260928 20:20:17.079280 12568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.1:34317 every 8 connection(s)
I20260928 20:20:17.079629 12441 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
I20260928 20:20:17.085198 12421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:17.080441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:49681 (local address 127.3.18.62:36239)
0928 20:20:17.080505 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:17.080507 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:17.080737 (+ 230us) server_negotiation.cc:407] Connection header received
0928 20:20:17.081223 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:17.081225 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:17.081238 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:17.081313 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:17.082040 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:17.082539 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:17.083677 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:17.083969 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:17.084029 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:17.084962 (+ 933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:20:17.084963 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:20:17.084964 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:20:17.084970 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:20:17.084984 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:20:17.084988 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:20:17.085036 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 20:20:17.085100 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 20:20:17.085133 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 20:20:17.085389 12571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:17.080392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36239 (local address 127.3.18.1:49681)
0928 20:20:17.080592 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:17.080633 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 20:20:17.081106 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:20:17.081381 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:20:17.081384 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:20:17.081392 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:20:17.081934 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:17.081938 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:17.082755 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:17.082756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:17.083579 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:17.083581 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:17.084060 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:17.084061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:17.084190 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:17.084821 (+ 631us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 20:20:17.084821 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:20:17.084852 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 20:20:17.084875 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:20:17.085035 (+ 160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:20:17.085037 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:20:17.085040 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 20:20:17.085088 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 20:20:17.085170 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":101,"threads_started":1}
I20260928 20:20:17.085964 12569 heartbeater.cc:347] Connected to a master server at 127.3.18.62:36239
I20260928 20:20:17.086083 12569 heartbeater.cc:464] Registering TS with master...
I20260928 20:20:17.086278 12569 heartbeater.cc:511] Master 127.3.18.62:36239 requested a full tablet report, sending...
I20260928 20:20:17.086894 12381 ts_manager.cc:194] Registered new tserver with Master: c8ed292aad9b4329a9b4e563c0144597 (127.3.18.1:34317)
I20260928 20:20:17.087566 12381 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.18.1:49681
I20260928 20:20:17.088742 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 12441
I20260928 20:20:17.088840 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-0/wal/instance
I20260928 20:20:17.089023 3144 external_mini_cluster.cc:1428] Setting key af225b286f5ddbb2d7673d510522a495
I20260928 20:20:17.090045 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.2:0
--local_ip_for_outbound_sockets=127.3.18.2
--webserver_interface=127.3.18.2
--webserver_port=0
--tserver_master_addrs=127.3.18.62:36239
--builtin_ntp_servers=127.3.18.20:36781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 20:20:17.173698 12572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:20:17.173924 12572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:20:17.173947 12572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:20:17.175297 12572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:20:17.175343 12572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:20:17.175369 12572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.2
I20260928 20:20:17.176934 12572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:36781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.18.2
--webserver_port=0
--tserver_master_addrs=127.3.18.62:36239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.2
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:20:17.177224 12572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:20:17.177557 12572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:20:17.183077 12577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:20:17.183092 12578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:20:17.183202 12572 server_base.cc:1034] running on GCE node
W20260928 20:20:17.183082 12580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:20:17.183553 12572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:20:17.183943 12572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:20:17.185101 12572 hybrid_clock.cc:648] HybridClock initialized: now 1790626817185081 us; error 44 us; skew 500 ppm
I20260928 20:20:17.186554 12572 webserver.cc:460] Webserver started at http://127.3.18.2:35149/ using document root <none> and password file <none>
I20260928 20:20:17.186898 12572 fs_manager.cc:359] Metadata directory not provided
I20260928 20:20:17.186980 12572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:20:17.187139 12572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:20:17.188339 12572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-1/data/instance:
uuid: "78bb23b11cf64f81b56c87787b09f91a"
format_stamp: "Formatted at 2026-09-28 20:20:17 on dist-test-slave-b83v"
server_key: "884f49a97cfcff7368fc5485900cbcd8"
server_key_iv: "e70f96d69895db1d2fd182a25ea4df90"
server_key_version: "encryptionkey@0"
I20260928 20:20:17.188730 12572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-1/wal/instance:
uuid: "78bb23b11cf64f81b56c87787b09f91a"
format_stamp: "Formatted at 2026-09-28 20:20:17 on dist-test-slave-b83v"
server_key: "884f49a97cfcff7368fc5485900cbcd8"
server_key_iv: "e70f96d69895db1d2fd182a25ea4df90"
server_key_version: "encryptionkey@0"
I20260928 20:20:17.190191 12572 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 20:20:17.191089 12586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:20:17.191442 12572 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:20:17.191576 12572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-1/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-1/wal
uuid: "78bb23b11cf64f81b56c87787b09f91a"
format_stamp: "Formatted at 2026-09-28 20:20:17 on dist-test-slave-b83v"
server_key: "884f49a97cfcff7368fc5485900cbcd8"
server_key_iv: "e70f96d69895db1d2fd182a25ea4df90"
server_key_version: "encryptionkey@0"
I20260928 20:20:17.191668 12572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:20:17.209794 12572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:20:17.210192 12572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:20:17.210382 12572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:20:17.210660 12572 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:20:17.211042 12572 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:20:17.211102 12572 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:20:17.211138 12572 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:20:17.211180 12572 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:20:17.217823 12572 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.2:36179
I20260928 20:20:17.217868 12699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.2:36179 every 8 connection(s)
I20260928 20:20:17.218205 12572 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
I20260928 20:20:17.223538 12421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:17.219001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:59043 (local address 127.3.18.62:36239)
0928 20:20:17.219065 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:17.219067 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:17.219372 (+ 305us) server_negotiation.cc:407] Connection header received
0928 20:20:17.219913 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:17.219914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:17.219925 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:17.219985 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:17.220680 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:17.221202 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:17.222206 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:17.222398 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:17.222467 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:17.223288 (+ 821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:20:17.223290 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:20:17.223290 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:20:17.223295 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:20:17.223312 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:20:17.223316 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:20:17.223370 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 20:20:17.223448 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 20:20:17.223474 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 20:20:17.223706 12702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:17.218989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36239 (local address 127.3.18.2:59043)
0928 20:20:17.219198 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:17.219234 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 20:20:17.219749 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:20:17.220043 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:20:17.220047 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:20:17.220055 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:20:17.220570 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:17.220574 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:17.221323 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:17.221324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:17.222116 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:17.222118 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:17.222484 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:17.222485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:17.222595 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:17.223150 (+ 555us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 20:20:17.223151 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:20:17.223180 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 20:20:17.223203 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:20:17.223379 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:20:17.223383 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:20:17.223385 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 20:20:17.223438 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 20:20:17.223518 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":103,"threads_started":1}
I20260928 20:20:17.224352 12700 heartbeater.cc:347] Connected to a master server at 127.3.18.62:36239
I20260928 20:20:17.224436 12700 heartbeater.cc:464] Registering TS with master...
I20260928 20:20:17.224617 12700 heartbeater.cc:511] Master 127.3.18.62:36239 requested a full tablet report, sending...
I20260928 20:20:17.225121 12381 ts_manager.cc:194] Registered new tserver with Master: 78bb23b11cf64f81b56c87787b09f91a (127.3.18.2:36179)
I20260928 20:20:17.225515 12381 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.18.2:59043
I20260928 20:20:17.225728 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 12572
I20260928 20:20:17.225816 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-1/wal/instance
I20260928 20:20:17.226029 3144 external_mini_cluster.cc:1428] Setting key a265638356d6d55942d67eafba2696f2
I20260928 20:20:17.226917 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.3:0
--local_ip_for_outbound_sockets=127.3.18.3
--webserver_interface=127.3.18.3
--webserver_port=0
--tserver_master_addrs=127.3.18.62:36239
--builtin_ntp_servers=127.3.18.20:36781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 20:20:17.312940 12703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:20:17.313185 12703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:20:17.313210 12703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:20:17.314507 12703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:20:17.314563 12703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:20:17.314589 12703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.3
I20260928 20:20:17.316107 12703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:36781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.18.3
--webserver_port=0
--tserver_master_addrs=127.3.18.62:36239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.3
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:20:17.316413 12703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:20:17.316754 12703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:20:17.322120 12708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:20:17.322131 12711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:20:17.322132 12709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:20:17.322198 12703 server_base.cc:1034] running on GCE node
I20260928 20:20:17.322546 12703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:20:17.322827 12703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:20:17.324015 12703 hybrid_clock.cc:648] HybridClock initialized: now 1790626817323996 us; error 28 us; skew 500 ppm
I20260928 20:20:17.325845 12703 webserver.cc:460] Webserver started at http://127.3.18.3:45199/ using document root <none> and password file <none>
I20260928 20:20:17.326141 12703 fs_manager.cc:359] Metadata directory not provided
I20260928 20:20:17.326220 12703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:20:17.326375 12703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:20:17.327514 12703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-2/data/instance:
uuid: "3c460dbe60b545e1937cc083f388b723"
format_stamp: "Formatted at 2026-09-28 20:20:17 on dist-test-slave-b83v"
server_key: "455683352893b2ef39e50fbfb9142952"
server_key_iv: "a4e1a16e53dae2c9f3d21611ec66101c"
server_key_version: "encryptionkey@0"
I20260928 20:20:17.327934 12703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-2/wal/instance:
uuid: "3c460dbe60b545e1937cc083f388b723"
format_stamp: "Formatted at 2026-09-28 20:20:17 on dist-test-slave-b83v"
server_key: "455683352893b2ef39e50fbfb9142952"
server_key_iv: "a4e1a16e53dae2c9f3d21611ec66101c"
server_key_version: "encryptionkey@0"
I20260928 20:20:17.329439 12703 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:20:17.330513 12717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:20:17.330766 12703 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:20:17.330888 12703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-2/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-2/wal
uuid: "3c460dbe60b545e1937cc083f388b723"
format_stamp: "Formatted at 2026-09-28 20:20:17 on dist-test-slave-b83v"
server_key: "455683352893b2ef39e50fbfb9142952"
server_key_iv: "a4e1a16e53dae2c9f3d21611ec66101c"
server_key_version: "encryptionkey@0"
I20260928 20:20:17.330983 12703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:20:17.353744 12703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:20:17.354106 12703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:20:17.354279 12703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:20:17.354535 12703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:20:17.354903 12703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:20:17.354965 12703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:20:17.355002 12703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:20:17.355049 12703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:20:17.361768 12703 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.3:39461
I20260928 20:20:17.361876 12830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.3:39461 every 8 connection(s)
I20260928 20:20:17.362305 12703 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
I20260928 20:20:17.368218 12421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:17.363300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:37001 (local address 127.3.18.62:36239)
0928 20:20:17.363435 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:17.363438 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:17.363618 (+ 180us) server_negotiation.cc:407] Connection header received
0928 20:20:17.364187 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:17.364189 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:17.364202 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:17.364270 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:17.364973 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:17.365506 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:17.366578 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:17.366823 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:17.366883 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:17.367919 (+ 1036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:20:17.367920 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:20:17.367921 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:20:17.367927 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:20:17.367944 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:20:17.367948 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:20:17.368005 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 20:20:17.368106 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 20:20:17.368139 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260928 20:20:17.368569 12833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:17.363263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36239 (local address 127.3.18.3:37001)
0928 20:20:17.363464 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:17.363509 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 20:20:17.364055 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:20:17.364335 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:20:17.364339 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:20:17.364349 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:20:17.364866 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:17.364870 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:17.365619 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:17.365620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:17.366486 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:17.366488 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:17.366932 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:17.366933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:17.367068 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:17.367696 (+ 628us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 20:20:17.367697 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:20:17.367733 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 20:20:17.367791 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:20:17.368021 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:20:17.368026 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:20:17.368029 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 20:20:17.368092 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 20:20:17.368197 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":87,"threads_started":1}
I20260928 20:20:17.369431 12831 heartbeater.cc:347] Connected to a master server at 127.3.18.62:36239
I20260928 20:20:17.369541 12831 heartbeater.cc:464] Registering TS with master...
I20260928 20:20:17.369700 12831 heartbeater.cc:511] Master 127.3.18.62:36239 requested a full tablet report, sending...
I20260928 20:20:17.370433 12381 ts_manager.cc:194] Registered new tserver with Master: 3c460dbe60b545e1937cc083f388b723 (127.3.18.3:39461)
I20260928 20:20:17.370798 12381 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.18.3:37001
I20260928 20:20:17.372526 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 12703
I20260928 20:20:17.372612 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-2/wal/instance
I20260928 20:20:17.372848 3144 external_mini_cluster.cc:1428] Setting key 6f7ca91f02b998c513cf2595933e0378
I20260928 20:20:17.374259 3144 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 20:20:17.376592 12421 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:20:17.375973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48252 (local address 127.3.18.62:36239)
0928 20:20:17.376047 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:17.376049 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:17.376172 (+ 123us) server_negotiation.cc:407] Connection header received
0928 20:20:17.376276 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:17.376277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:17.376291 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:17.376345 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:17.376485 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48252: BlockingRecv error: recv got EOF from 127.0.0.1:48252 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 20:20:17.376646 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 12441
I20260928 20:20:17.383419 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 12572
I20260928 20:20:17.388002 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 12703
I20260928 20:20:17.392700 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 12350
2026-09-28T20:20:17Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (651 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T20:20:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:20:17Z Disabled control of system clock
I20260928 20:20:17.420521 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790626764250411-3144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790626764250411-3144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790626764250411-3144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:40039
--webserver_interface=127.3.18.62
--webserver_port=0
--builtin_ntp_servers=127.3.18.20:34721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.18.62:40039
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 20:20:17.506831 12842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:20:17.507071 12842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:20:17.507097 12842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:20:17.508369 12842 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:20:17.508419 12842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:20:17.508431 12842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:20:17.508445 12842 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:20:17.508457 12842 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:20:17.509894 12842 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:34721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790626764250411-3144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790626764250411-3144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.18.62:40039
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:40039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.18.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790626764250411-3144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:20:17.510142 12842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:20:17.510439 12842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:20:17.516165 12850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:20:17.516150 12848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:20:17.516150 12847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:20:17.516376 12842 server_base.cc:1034] running on GCE node
I20260928 20:20:17.516652 12842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:20:17.516986 12842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:20:17.518162 12842 hybrid_clock.cc:648] HybridClock initialized: now 1790626817518146 us; error 59 us; skew 500 ppm
I20260928 20:20:17.519913 12842 webserver.cc:460] Webserver started at http://127.3.18.62:39103/ using document root <none> and password file <none>
I20260928 20:20:17.520210 12842 fs_manager.cc:359] Metadata directory not provided
I20260928 20:20:17.520270 12842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:20:17.520466 12842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:20:17.521720 12842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790626764250411-3144-0/minicluster-data/master-0/data/instance:
uuid: "3345ecbdf9fe49e9be2ee1f3bd51ce09"
format_stamp: "Formatted at 2026-09-28 20:20:17 on dist-test-slave-b83v"
server_key: "546e86eb56007a66c1db26e97db6d28e"
server_key_iv: "6973bb37156aca0342d0ba322fd71f4a"
server_key_version: "encryptionkey@0"
I20260928 20:20:17.522156 12842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790626764250411-3144-0/minicluster-data/master-0/wal/instance:
uuid: "3345ecbdf9fe49e9be2ee1f3bd51ce09"
format_stamp: "Formatted at 2026-09-28 20:20:17 on dist-test-slave-b83v"
server_key: "546e86eb56007a66c1db26e97db6d28e"
server_key_iv: "6973bb37156aca0342d0ba322fd71f4a"
server_key_version: "encryptionkey@0"
I20260928 20:20:17.523720 12842 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260928 20:20:17.524693 12856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:20:17.524923 12842 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:20:17.525033 12842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790626764250411-3144-0/minicluster-data/master-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790626764250411-3144-0/minicluster-data/master-0/wal
uuid: "3345ecbdf9fe49e9be2ee1f3bd51ce09"
format_stamp: "Formatted at 2026-09-28 20:20:17 on dist-test-slave-b83v"
server_key: "546e86eb56007a66c1db26e97db6d28e"
server_key_iv: "6973bb37156aca0342d0ba322fd71f4a"
server_key_version: "encryptionkey@0"
I20260928 20:20:17.525125 12842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790626764250411-3144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790626764250411-3144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790626764250411-3144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:20:17.528713 12842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:20:17.529021 12842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:20:17.529170 12842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:20:17.533238 12842 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.62:40039
I20260928 20:20:17.533332 12908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.62:40039 every 8 connection(s)
I20260928 20:20:17.533667 12842 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
I20260928 20:20:17.534482 12909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:20:17.536720 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 12842
I20260928 20:20:17.536854 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790626764250411-3144-0/minicluster-data/master-0/wal/instance
I20260928 20:20:17.537101 3144 external_mini_cluster.cc:1428] Setting key 7e44acc17c2a504cebf10cc3579cf8a4
I20260928 20:20:17.538513 12909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3345ecbdf9fe49e9be2ee1f3bd51ce09: Bootstrap starting.
I20260928 20:20:17.539237 12909 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3345ecbdf9fe49e9be2ee1f3bd51ce09: Neither blocks nor log segments found. Creating new log.
I20260928 20:20:17.539592 12909 log.cc:824] T 00000000000000000000000000000000 P 3345ecbdf9fe49e9be2ee1f3bd51ce09: Log is configured to *not* fsync() on all Append() calls
I20260928 20:20:17.540251 12912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:17.537822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33632 (local address 127.3.18.62:40039)
0928 20:20:17.538043 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:17.538051 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:17.538074 (+ 23us) server_negotiation.cc:407] Connection header received
0928 20:20:17.538491 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:17.538498 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:17.538539 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:17.538703 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:17.539742 (+ 1039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:20:17.539746 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:20:17.539747 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:20:17.539751 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:20:17.539825 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:20:17.539839 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:20:17.539923 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 20:20:17.539995 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 20:20:17.540056 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":93,"threads_started":1}
I20260928 20:20:17.540541 12909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3345ecbdf9fe49e9be2ee1f3bd51ce09: No bootstrap required, opened a new log
I20260928 20:20:17.542110 12909 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3345ecbdf9fe49e9be2ee1f3bd51ce09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3345ecbdf9fe49e9be2ee1f3bd51ce09" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 40039 } }
I20260928 20:20:17.542265 12909 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3345ecbdf9fe49e9be2ee1f3bd51ce09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:20:17.542289 12909 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3345ecbdf9fe49e9be2ee1f3bd51ce09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3345ecbdf9fe49e9be2ee1f3bd51ce09, State: Initialized, Role: FOLLOWER
I20260928 20:20:17.542409 12909 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3345ecbdf9fe49e9be2ee1f3bd51ce09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3345ecbdf9fe49e9be2ee1f3bd51ce09" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 40039 } }
I20260928 20:20:17.542495 12909 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3345ecbdf9fe49e9be2ee1f3bd51ce09 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:20:17.542518 12909 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3345ecbdf9fe49e9be2ee1f3bd51ce09 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:20:17.542548 12909 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3345ecbdf9fe49e9be2ee1f3bd51ce09 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:20:17.543381 12909 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3345ecbdf9fe49e9be2ee1f3bd51ce09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3345ecbdf9fe49e9be2ee1f3bd51ce09" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 40039 } }
I20260928 20:20:17.543471 12909 leader_election.cc:302] T 00000000000000000000000000000000 P 3345ecbdf9fe49e9be2ee1f3bd51ce09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3345ecbdf9fe49e9be2ee1f3bd51ce09; no voters:
I20260928 20:20:17.543596 12909 leader_election.cc:288] T 00000000000000000000000000000000 P 3345ecbdf9fe49e9be2ee1f3bd51ce09 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:20:17.543722 12914 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3345ecbdf9fe49e9be2ee1f3bd51ce09 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:20:17.543913 12914 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3345ecbdf9fe49e9be2ee1f3bd51ce09 [term 1 LEADER]: Becoming Leader. State: Replica: 3345ecbdf9fe49e9be2ee1f3bd51ce09, State: Running, Role: LEADER
I20260928 20:20:17.544035 12914 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3345ecbdf9fe49e9be2ee1f3bd51ce09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3345ecbdf9fe49e9be2ee1f3bd51ce09" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 40039 } }
I20260928 20:20:17.544127 12909 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3345ecbdf9fe49e9be2ee1f3bd51ce09 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:20:17.544538 12916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3345ecbdf9fe49e9be2ee1f3bd51ce09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3345ecbdf9fe49e9be2ee1f3bd51ce09. Latest consensus state: current_term: 1 leader_uuid: "3345ecbdf9fe49e9be2ee1f3bd51ce09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3345ecbdf9fe49e9be2ee1f3bd51ce09" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 40039 } } }
I20260928 20:20:17.544580 12915 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3345ecbdf9fe49e9be2ee1f3bd51ce09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3345ecbdf9fe49e9be2ee1f3bd51ce09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3345ecbdf9fe49e9be2ee1f3bd51ce09" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 40039 } } }
I20260928 20:20:17.544771 12916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3345ecbdf9fe49e9be2ee1f3bd51ce09 [sys.catalog]: This master's current role is: LEADER
I20260928 20:20:17.544792 12915 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3345ecbdf9fe49e9be2ee1f3bd51ce09 [sys.catalog]: This master's current role is: LEADER
I20260928 20:20:17.545182 12923 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:20:17.545789 12923 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:20:17.547869 12923 catalog_manager.cc:1347] Generated new cluster ID: 0d1c39554abd4184bf4ee5c5bfd53569
I20260928 20:20:17.547935 12923 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:20:17.562108 12923 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:20:17.562397 12923 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:20:17.566612 12923 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3345ecbdf9fe49e9be2ee1f3bd51ce09: Generated new TSK 0
I20260928 20:20:17.566871 12923 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 20:20:17.574020 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.1:0
--local_ip_for_outbound_sockets=127.3.18.1
--webserver_interface=127.3.18.1
--webserver_port=0
--tserver_master_addrs=127.3.18.62:40039
--builtin_ntp_servers=127.3.18.20:34721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 20:20:17.676852 12933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:20:17.677122 12933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:20:17.677163 12933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:20:17.678370 12933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:20:17.678440 12933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:20:17.678486 12933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.1
I20260928 20:20:17.679888 12933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:34721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.18.1
--webserver_port=0
--tserver_master_addrs=127.3.18.62:40039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.1
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:20:17.680197 12933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:20:17.680522 12933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:20:17.685688 12938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:20:17.685647 12939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:20:17.685706 12941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:20:17.685858 12933 server_base.cc:1034] running on GCE node
I20260928 20:20:17.686152 12933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:20:17.686440 12933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:20:17.687629 12933 hybrid_clock.cc:648] HybridClock initialized: now 1790626817687616 us; error 28 us; skew 500 ppm
I20260928 20:20:17.694135 12933 webserver.cc:460] Webserver started at http://127.3.18.1:42441/ using document root <none> and password file <none>
I20260928 20:20:17.694473 12933 fs_manager.cc:359] Metadata directory not provided
I20260928 20:20:17.694527 12933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:20:17.694677 12933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:20:17.695890 12933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-0/data/instance:
uuid: "f14f1bbadf67498d80f81e0b98514c84"
format_stamp: "Formatted at 2026-09-28 20:20:17 on dist-test-slave-b83v"
server_key: "ed1f1ad2097015d4cef22dd3f7481033"
server_key_iv: "bba8640d006be79f9219c3a0e1146208"
server_key_version: "encryptionkey@0"
I20260928 20:20:17.696278 12933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-0/wal/instance:
uuid: "f14f1bbadf67498d80f81e0b98514c84"
format_stamp: "Formatted at 2026-09-28 20:20:17 on dist-test-slave-b83v"
server_key: "ed1f1ad2097015d4cef22dd3f7481033"
server_key_iv: "bba8640d006be79f9219c3a0e1146208"
server_key_version: "encryptionkey@0"
I20260928 20:20:17.697688 12933 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 20:20:17.698520 12947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:20:17.698750 12933 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:20:17.698876 12933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-0/wal
uuid: "f14f1bbadf67498d80f81e0b98514c84"
format_stamp: "Formatted at 2026-09-28 20:20:17 on dist-test-slave-b83v"
server_key: "ed1f1ad2097015d4cef22dd3f7481033"
server_key_iv: "bba8640d006be79f9219c3a0e1146208"
server_key_version: "encryptionkey@0"
I20260928 20:20:17.698971 12933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:20:17.704574 12933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:20:17.704908 12933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:20:17.705065 12933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:20:17.705313 12933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:20:17.705688 12933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:20:17.705736 12933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:20:17.705772 12933 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:20:17.705813 12933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:20:17.714416 12933 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.1:39597
I20260928 20:20:17.714555 13060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.1:39597 every 8 connection(s)
I20260928 20:20:17.714931 12933 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
I20260928 20:20:17.718480 12912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:17.715853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:57805 (local address 127.3.18.62:40039)
0928 20:20:17.715936 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:17.715938 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:17.716201 (+ 263us) server_negotiation.cc:407] Connection header received
0928 20:20:17.716701 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:17.716703 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:17.716711 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:17.716830 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:17.718121 (+ 1291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:20:17.718123 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:20:17.718124 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:20:17.718131 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:20:17.718159 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:20:17.718165 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:20:17.718266 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 20:20:17.718342 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 20:20:17.718364 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 20:20:17.718576 13063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:17.715845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40039 (local address 127.3.18.1:57805)
0928 20:20:17.716058 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:17.716100 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 20:20:17.716586 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:20:17.716894 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:20:17.716898 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:20:17.716906 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:20:17.717937 (+ 1031us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 20:20:17.717937 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:20:17.717968 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 20:20:17.717999 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:20:17.718262 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:20:17.718265 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:20:17.718268 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 20:20:17.718325 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 20:20:17.718378 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":113,"threads_started":1}
I20260928 20:20:17.719103 13061 heartbeater.cc:347] Connected to a master server at 127.3.18.62:40039
I20260928 20:20:17.719206 13061 heartbeater.cc:464] Registering TS with master...
I20260928 20:20:17.719367 13061 heartbeater.cc:511] Master 127.3.18.62:40039 requested a full tablet report, sending...
I20260928 20:20:17.719944 12873 ts_manager.cc:194] Registered new tserver with Master: f14f1bbadf67498d80f81e0b98514c84 (127.3.18.1:39597)
I20260928 20:20:17.720702 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 12933
I20260928 20:20:17.720781 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-0/wal/instance
I20260928 20:20:17.720989 3144 external_mini_cluster.cc:1428] Setting key c73530f8235a3ffee4d807f9dd623a19
I20260928 20:20:17.721886 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.2:0
--local_ip_for_outbound_sockets=127.3.18.2
--webserver_interface=127.3.18.2
--webserver_port=0
--tserver_master_addrs=127.3.18.62:40039
--builtin_ntp_servers=127.3.18.20:34721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 20:20:17.811390 13064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:20:17.811596 13064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:20:17.811617 13064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:20:17.812767 13064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:20:17.812805 13064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:20:17.812829 13064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.2
I20260928 20:20:17.814165 13064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:34721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.18.2
--webserver_port=0
--tserver_master_addrs=127.3.18.62:40039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.2
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:20:17.814368 13064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:20:17.814618 13064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:20:17.819742 13069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:20:17.819744 13072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:20:17.819746 13070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:20:17.819855 13064 server_base.cc:1034] running on GCE node
I20260928 20:20:17.820245 13064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:20:17.820498 13064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:20:17.821696 13064 hybrid_clock.cc:648] HybridClock initialized: now 1790626817821667 us; error 44 us; skew 500 ppm
I20260928 20:20:17.823371 13064 webserver.cc:460] Webserver started at http://127.3.18.2:38733/ using document root <none> and password file <none>
I20260928 20:20:17.823678 13064 fs_manager.cc:359] Metadata directory not provided
I20260928 20:20:17.823817 13064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:20:17.823984 13064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:20:17.825213 13064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-1/data/instance:
uuid: "9b38780100164e489ebe820e837691f2"
format_stamp: "Formatted at 2026-09-28 20:20:17 on dist-test-slave-b83v"
server_key: "6e930819169e89cb41a64e62e40a83df"
server_key_iv: "eaa518a85aa496cbc3a99fdf410d9bd2"
server_key_version: "encryptionkey@0"
I20260928 20:20:17.825639 13064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-1/wal/instance:
uuid: "9b38780100164e489ebe820e837691f2"
format_stamp: "Formatted at 2026-09-28 20:20:17 on dist-test-slave-b83v"
server_key: "6e930819169e89cb41a64e62e40a83df"
server_key_iv: "eaa518a85aa496cbc3a99fdf410d9bd2"
server_key_version: "encryptionkey@0"
I20260928 20:20:17.827104 13064 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:20:17.828155 13078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:20:17.828465 13064 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 20:20:17.828577 13064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-1/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-1/wal
uuid: "9b38780100164e489ebe820e837691f2"
format_stamp: "Formatted at 2026-09-28 20:20:17 on dist-test-slave-b83v"
server_key: "6e930819169e89cb41a64e62e40a83df"
server_key_iv: "eaa518a85aa496cbc3a99fdf410d9bd2"
server_key_version: "encryptionkey@0"
I20260928 20:20:17.828673 13064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:20:17.832479 13064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:20:17.832782 13064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:20:17.832942 13064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:20:17.833177 13064 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:20:17.833529 13064 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:20:17.833590 13064 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:20:17.833633 13064 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:20:17.833664 13064 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:20:17.840114 13064 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.2:40827
I20260928 20:20:17.840152 13191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.2:40827 every 8 connection(s)
I20260928 20:20:17.840547 13064 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
I20260928 20:20:17.844426 12912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:17.841438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:40157 (local address 127.3.18.62:40039)
0928 20:20:17.841506 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:17.841508 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:17.841822 (+ 314us) server_negotiation.cc:407] Connection header received
0928 20:20:17.842419 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:17.842421 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:17.842427 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:17.842498 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:17.844069 (+ 1571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:20:17.844071 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:20:17.844071 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:20:17.844075 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:20:17.844092 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:20:17.844096 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:20:17.844195 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 20:20:17.844335 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 20:20:17.844352 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 20:20:17.844663 13194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:17.841440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40039 (local address 127.3.18.2:40157)
0928 20:20:17.841678 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:17.841718 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 20:20:17.842253 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:20:17.842595 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:20:17.842599 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:20:17.842608 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:20:17.843900 (+ 1292us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 20:20:17.843901 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:20:17.843942 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 20:20:17.843978 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:20:17.844210 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:20:17.844215 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:20:17.844218 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 20:20:17.844299 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 20:20:17.844385 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":92,"threads_started":1}
I20260928 20:20:17.845283 13192 heartbeater.cc:347] Connected to a master server at 127.3.18.62:40039
I20260928 20:20:17.845387 13192 heartbeater.cc:464] Registering TS with master...
I20260928 20:20:17.845557 13192 heartbeater.cc:511] Master 127.3.18.62:40039 requested a full tablet report, sending...
I20260928 20:20:17.846040 12873 ts_manager.cc:194] Registered new tserver with Master: 9b38780100164e489ebe820e837691f2 (127.3.18.2:40827)
I20260928 20:20:17.847898 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 13064
I20260928 20:20:17.848007 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-1/wal/instance
I20260928 20:20:17.848248 3144 external_mini_cluster.cc:1428] Setting key 44b922333cb4a3e16b8c6448ce20a9f5
I20260928 20:20:17.849269 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.3:0
--local_ip_for_outbound_sockets=127.3.18.3
--webserver_interface=127.3.18.3
--webserver_port=0
--tserver_master_addrs=127.3.18.62:40039
--builtin_ntp_servers=127.3.18.20:34721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 20:20:17.942444 13195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:20:17.942703 13195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:20:17.942730 13195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:20:17.944424 13195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:20:17.944473 13195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:20:17.944504 13195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.3
I20260928 20:20:17.946275 13195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:34721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.18.3
--webserver_port=0
--tserver_master_addrs=127.3.18.62:40039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.3
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:20:17.946594 13195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:20:17.946945 13195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:20:17.952786 13200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:20:17.952898 13203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:20:17.952829 13195 server_base.cc:1034] running on GCE node
W20260928 20:20:17.952790 13201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:20:17.953353 13195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:20:17.953626 13195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:20:17.954798 13195 hybrid_clock.cc:648] HybridClock initialized: now 1790626817954798 us; error 43 us; skew 500 ppm
I20260928 20:20:17.956223 13195 webserver.cc:460] Webserver started at http://127.3.18.3:46435/ using document root <none> and password file <none>
I20260928 20:20:17.956526 13195 fs_manager.cc:359] Metadata directory not provided
I20260928 20:20:17.956601 13195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:20:17.956756 13195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:20:17.957859 13195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-2/data/instance:
uuid: "c3be76af5ca444ea89ca44322f85c9c1"
format_stamp: "Formatted at 2026-09-28 20:20:17 on dist-test-slave-b83v"
server_key: "76821b74a47b65e472a6677b8f80cca6"
server_key_iv: "ebbf991ded6448f0c35b8e38a7514534"
server_key_version: "encryptionkey@0"
I20260928 20:20:17.958235 13195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-2/wal/instance:
uuid: "c3be76af5ca444ea89ca44322f85c9c1"
format_stamp: "Formatted at 2026-09-28 20:20:17 on dist-test-slave-b83v"
server_key: "76821b74a47b65e472a6677b8f80cca6"
server_key_iv: "ebbf991ded6448f0c35b8e38a7514534"
server_key_version: "encryptionkey@0"
I20260928 20:20:17.959591 13195 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 20:20:17.960493 13209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:20:17.960672 13195 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:20:17.960775 13195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-2/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-2/wal
uuid: "c3be76af5ca444ea89ca44322f85c9c1"
format_stamp: "Formatted at 2026-09-28 20:20:17 on dist-test-slave-b83v"
server_key: "76821b74a47b65e472a6677b8f80cca6"
server_key_iv: "ebbf991ded6448f0c35b8e38a7514534"
server_key_version: "encryptionkey@0"
I20260928 20:20:17.960862 13195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:20:17.964479 13195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:20:17.964743 13195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:20:17.964881 13195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:20:17.965078 13195 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:20:17.965380 13195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:20:17.965431 13195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:20:17.965471 13195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:20:17.965499 13195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:20:17.971581 13195 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.3:43933
I20260928 20:20:17.971664 13322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.3:43933 every 8 connection(s)
I20260928 20:20:17.972054 13195 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
I20260928 20:20:17.974779 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 13195
I20260928 20:20:17.974859 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790626764250411-3144-0/minicluster-data/ts-2/wal/instance
I20260928 20:20:17.975064 3144 external_mini_cluster.cc:1428] Setting key 5ca8315e8e514fce588c4d51a5aae68c
I20260928 20:20:17.975312 12912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:17.972871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:50973 (local address 127.3.18.62:40039)
0928 20:20:17.972954 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:17.972956 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:17.973209 (+ 253us) server_negotiation.cc:407] Connection header received
0928 20:20:17.973675 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:17.973677 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:17.973683 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:17.973755 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:17.974990 (+ 1235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:20:17.974991 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:20:17.974992 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:20:17.974997 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:20:17.975015 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:20:17.975020 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:20:17.975078 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 20:20:17.975213 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 20:20:17.975229 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 20:20:17.975499 13325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:17.972803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40039 (local address 127.3.18.3:50973)
0928 20:20:17.973019 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:17.973062 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 20:20:17.973576 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:20:17.973815 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:20:17.973818 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:20:17.973827 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:20:17.974760 (+ 933us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 20:20:17.974761 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:20:17.974791 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 20:20:17.974822 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:20:17.975086 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:20:17.975092 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:20:17.975096 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:20:17.975176 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 20:20:17.975260 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":105,"threads_started":1}
I20260928 20:20:17.975997 13323 heartbeater.cc:347] Connected to a master server at 127.3.18.62:40039
I20260928 20:20:17.976102 13323 heartbeater.cc:464] Registering TS with master...
I20260928 20:20:17.976259 13323 heartbeater.cc:511] Master 127.3.18.62:40039 requested a full tablet report, sending...
I20260928 20:20:17.976763 12872 ts_manager.cc:194] Registered new tserver with Master: c3be76af5ca444ea89ca44322f85c9c1 (127.3.18.3:43933)
I20260928 20:20:17.987195 3144 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 20:20:17.989238 12912 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:20:17.988719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33646 (local address 127.3.18.62:40039)
0928 20:20:17.988784 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:17.988786 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:17.988931 (+ 145us) server_negotiation.cc:407] Connection header received
0928 20:20:17.988980 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:17.988981 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:17.988987 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:17.989038 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:17.989138 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33646: BlockingRecv error: recv got EOF from 127.0.0.1:33646 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 20:20:17.989369 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 12933
I20260928 20:20:17.993966 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 13064
I20260928 20:20:17.998380 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 13195
I20260928 20:20:18.002959 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 12842
2026-09-28T20:20:18Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (610 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:20:18 dist-test-slave-b83v krb5kdc[13333](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:20:18 dist-test-slave-b83v krb5kdc[13333](info): set up 2 sockets
Sep 28 20:20:18 dist-test-slave-b83v krb5kdc[13333](info): commencing operation
krb5kdc: starting...
W20260928 20:20:19.049111 3144 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Sep 28 20:20:19 dist-test-slave-b83v krb5kdc[13333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:20:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:20:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.18.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab.
Entry for principal kudu/127.3.18.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab.
I20260928 20:20:19.184605 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:41665
--webserver_interface=127.3.18.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:39535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.18.62:41665
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:20:19.270303 13346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:20:19.270538 13346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:20:19.270569 13346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:20:19.271876 13346 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:20:19.271924 13346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:20:19.271938 13346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:20:19.271951 13346 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:20:19.271963 13346 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:20:19.273577 13346 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:39535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.18.62:41665
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:41665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.18.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:20:19.273901 13346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:20:19.274163 13346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:20:19.279755 13352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:20:19.279944 13346 server_base.cc:1034] running on GCE node
W20260928 20:20:19.279778 13351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:20:19.279778 13354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:20:19.280650 13346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:20:19.281061 13346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:20:19.282287 13346 hybrid_clock.cc:648] HybridClock initialized: now 1790626819282249 us; error 61 us; skew 500 ppm
Sep 28 20:20:19 dist-test-slave-b83v krb5kdc[13333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626819, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:20:19.285693 13346 init.cc:377] Logged in from keytab as kudu/127.3.18.62@KRBTEST.COM (short username kudu)
I20260928 20:20:19.286222 13346 webserver.cc:460] Webserver started at http://127.3.18.62:40649/ using document root <none> and password file <none>
I20260928 20:20:19.286531 13346 fs_manager.cc:359] Metadata directory not provided
I20260928 20:20:19.286590 13346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:20:19.286759 13346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:20:19.288086 13346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/minicluster-data/master-0/data/instance:
uuid: "8455ccb20cf14d4cabd1a394435e7c34"
format_stamp: "Formatted at 2026-09-28 20:20:19 on dist-test-slave-b83v"
server_key: "dc22a3b99f9847d129b59b3b781ce7a4"
server_key_iv: "d001c25aac2e00e189422e95205b754d"
server_key_version: "encryptionkey@0"
I20260928 20:20:19.288515 13346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/minicluster-data/master-0/wal/instance:
uuid: "8455ccb20cf14d4cabd1a394435e7c34"
format_stamp: "Formatted at 2026-09-28 20:20:19 on dist-test-slave-b83v"
server_key: "dc22a3b99f9847d129b59b3b781ce7a4"
server_key_iv: "d001c25aac2e00e189422e95205b754d"
server_key_version: "encryptionkey@0"
I20260928 20:20:19.290180 13346 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 20:20:19.291082 13361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:20:19.291405 13346 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:20:19.291518 13346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/minicluster-data/master-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/minicluster-data/master-0/wal
uuid: "8455ccb20cf14d4cabd1a394435e7c34"
format_stamp: "Formatted at 2026-09-28 20:20:19 on dist-test-slave-b83v"
server_key: "dc22a3b99f9847d129b59b3b781ce7a4"
server_key_iv: "d001c25aac2e00e189422e95205b754d"
server_key_version: "encryptionkey@0"
I20260928 20:20:19.291599 13346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:20:19.312041 13346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:20:19.314392 13346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:20:19.314558 13346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:20:19.319051 13346 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.62:41665
I20260928 20:20:19.319108 13413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.62:41665 every 8 connection(s)
I20260928 20:20:19.319535 13346 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
I20260928 20:20:19.320437 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 13346
I20260928 20:20:19.320570 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/minicluster-data/master-0/wal/instance
I20260928 20:20:19.320786 3144 external_mini_cluster.cc:1428] Setting key f6088993b5b26dfb039fb1115236cd8e
I20260928 20:20:19.321058 13414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:20:19.324378 13414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8455ccb20cf14d4cabd1a394435e7c34: Bootstrap starting.
I20260928 20:20:19.325002 13414 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8455ccb20cf14d4cabd1a394435e7c34: Neither blocks nor log segments found. Creating new log.
I20260928 20:20:19.325323 13414 log.cc:824] T 00000000000000000000000000000000 P 8455ccb20cf14d4cabd1a394435e7c34: Log is configured to *not* fsync() on all Append() calls
I20260928 20:20:19.326098 13414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8455ccb20cf14d4cabd1a394435e7c34: No bootstrap required, opened a new log
Sep 28 20:20:19 dist-test-slave-b83v krb5kdc[13333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:20:19.328467 13414 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8455ccb20cf14d4cabd1a394435e7c34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8455ccb20cf14d4cabd1a394435e7c34" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 41665 } }
I20260928 20:20:19.328684 13414 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8455ccb20cf14d4cabd1a394435e7c34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:20:19.328732 13414 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8455ccb20cf14d4cabd1a394435e7c34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8455ccb20cf14d4cabd1a394435e7c34, State: Initialized, Role: FOLLOWER
I20260928 20:20:19.328882 13414 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8455ccb20cf14d4cabd1a394435e7c34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8455ccb20cf14d4cabd1a394435e7c34" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 41665 } }
I20260928 20:20:19.328990 13414 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8455ccb20cf14d4cabd1a394435e7c34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:20:19.329027 13414 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8455ccb20cf14d4cabd1a394435e7c34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:20:19.329088 13414 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8455ccb20cf14d4cabd1a394435e7c34 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:20:19.329835 13414 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8455ccb20cf14d4cabd1a394435e7c34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8455ccb20cf14d4cabd1a394435e7c34" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 41665 } }
I20260928 20:20:19.329928 13414 leader_election.cc:302] T 00000000000000000000000000000000 P 8455ccb20cf14d4cabd1a394435e7c34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8455ccb20cf14d4cabd1a394435e7c34; no voters:
I20260928 20:20:19.330134 13414 leader_election.cc:288] T 00000000000000000000000000000000 P 8455ccb20cf14d4cabd1a394435e7c34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:20:19.330192 13419 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8455ccb20cf14d4cabd1a394435e7c34 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:20:19.330416 13414 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8455ccb20cf14d4cabd1a394435e7c34 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:20:19.330435 13419 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8455ccb20cf14d4cabd1a394435e7c34 [term 1 LEADER]: Becoming Leader. State: Replica: 8455ccb20cf14d4cabd1a394435e7c34, State: Running, Role: LEADER
I20260928 20:20:19.330538 13419 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8455ccb20cf14d4cabd1a394435e7c34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8455ccb20cf14d4cabd1a394435e7c34" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 41665 } }
I20260928 20:20:19.330961 13421 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8455ccb20cf14d4cabd1a394435e7c34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8455ccb20cf14d4cabd1a394435e7c34. Latest consensus state: current_term: 1 leader_uuid: "8455ccb20cf14d4cabd1a394435e7c34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8455ccb20cf14d4cabd1a394435e7c34" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 41665 } } }
I20260928 20:20:19.331102 13421 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8455ccb20cf14d4cabd1a394435e7c34 [sys.catalog]: This master's current role is: LEADER
I20260928 20:20:19.331368 13428 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:20:19.331940 13428 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:20:19.332861 13420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8455ccb20cf14d4cabd1a394435e7c34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8455ccb20cf14d4cabd1a394435e7c34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8455ccb20cf14d4cabd1a394435e7c34" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 41665 } } }
I20260928 20:20:19.333043 13420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8455ccb20cf14d4cabd1a394435e7c34 [sys.catalog]: This master's current role is: LEADER
I20260928 20:20:19.333089 13417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:19.321382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45820 (local address 127.3.18.62:41665)
0928 20:20:19.321577 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:19.321586 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:19.321622 (+ 36us) server_negotiation.cc:407] Connection header received
0928 20:20:19.322335 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:19.322343 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:19.322404 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:19.322579 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:19.323283 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:19.324287 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:19.325130 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:19.325605 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:19.325726 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:19.328515 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:20:19.328520 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:20:19.328520 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:20:19.328531 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:20:19.330725 (+ 2194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:20:19.331236 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:20:19.331239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:20:19.331240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:20:19.331372 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:20:19.331631 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:20:19.331632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:20:19.331633 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:20:19.332009 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:20:19.332176 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 20:20:19.332723 (+ 547us) server_negotiation.cc:299] Negotiation successful
0928 20:20:19.332888 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":90,"threads_started":1}
I20260928 20:20:19.333880 13428 catalog_manager.cc:1347] Generated new cluster ID: d2f4f4966b914f78b70d7e7567736ee8
I20260928 20:20:19.333945 13428 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:20:19.391263 13428 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:20:19.392009 13428 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:20:19.401351 13428 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8455ccb20cf14d4cabd1a394435e7c34: Generated new TSK 0
I20260928 20:20:19.401579 13428 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.18.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab.
Entry for principal kudu/127.3.18.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab.
I20260928 20:20:19.436236 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.1:0
--local_ip_for_outbound_sockets=127.3.18.1
--webserver_interface=127.3.18.1
--webserver_port=0
--tserver_master_addrs=127.3.18.62:41665
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab
--principal=kudu/127.3.18.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:39535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:20:19.534004 13440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:20:19.534230 13440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:20:19.534256 13440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:20:19.535473 13440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:20:19.535521 13440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:20:19.535548 13440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.1
I20260928 20:20:19.537114 13440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:39535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab
--principal=kudu/127.3.18.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.18.1
--webserver_port=0
--tserver_master_addrs=127.3.18.62:41665
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.1
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:20:19.537400 13440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:20:19.537647 13440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:20:19.543040 13446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:20:19.543042 13448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:20:19.543332 13440 server_base.cc:1034] running on GCE node
W20260928 20:20:19.543071 13445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:20:19.543610 13440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:20:19.543917 13440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:20:19.545061 13440 hybrid_clock.cc:648] HybridClock initialized: now 1790626819545049 us; error 29 us; skew 500 ppm
Sep 28 20:20:19 dist-test-slave-b83v krb5kdc[13333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626819, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:20:19.547885 13440 init.cc:377] Logged in from keytab as kudu/127.3.18.1@KRBTEST.COM (short username kudu)
I20260928 20:20:19.548398 13440 webserver.cc:460] Webserver started at http://127.3.18.1:45533/ using document root <none> and password file <none>
I20260928 20:20:19.548595 13440 fs_manager.cc:359] Metadata directory not provided
I20260928 20:20:19.548637 13440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:20:19.548733 13440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:20:19.549772 13440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/minicluster-data/ts-0/data/instance:
uuid: "bbfa42bfe72742cea70d82f899336911"
format_stamp: "Formatted at 2026-09-28 20:20:19 on dist-test-slave-b83v"
server_key: "97af36cabf28ca944e4d21a1da4a0dba"
server_key_iv: "02cf67abf092deac7a4dcfba81dfca34"
server_key_version: "encryptionkey@0"
I20260928 20:20:19.550093 13440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/minicluster-data/ts-0/wal/instance:
uuid: "bbfa42bfe72742cea70d82f899336911"
format_stamp: "Formatted at 2026-09-28 20:20:19 on dist-test-slave-b83v"
server_key: "97af36cabf28ca944e4d21a1da4a0dba"
server_key_iv: "02cf67abf092deac7a4dcfba81dfca34"
server_key_version: "encryptionkey@0"
I20260928 20:20:19.551438 13440 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:20:19.552428 13455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:20:19.552662 13440 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 20:20:19.552767 13440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/minicluster-data/ts-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/minicluster-data/ts-0/wal
uuid: "bbfa42bfe72742cea70d82f899336911"
format_stamp: "Formatted at 2026-09-28 20:20:19 on dist-test-slave-b83v"
server_key: "97af36cabf28ca944e4d21a1da4a0dba"
server_key_iv: "02cf67abf092deac7a4dcfba81dfca34"
server_key_version: "encryptionkey@0"
I20260928 20:20:19.552899 13440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:20:19.569630 13440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:20:19.572333 13440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:20:19.572501 13440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:20:19.572742 13440 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:20:19.573110 13440 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:20:19.573167 13440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:20:19.573200 13440 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:20:19.573230 13440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:20:19.579231 13440 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.1:39141
I20260928 20:20:19.579330 13568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.1:39141 every 8 connection(s)
I20260928 20:20:19.579662 13440 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
I20260928 20:20:19.582508 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 13440
I20260928 20:20:19.582603 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/minicluster-data/ts-0/wal/instance
I20260928 20:20:19.582824 3144 external_mini_cluster.cc:1428] Setting key bd851ce09502e0be64670b8bf0602790
Sep 28 20:20:19 dist-test-slave-b83v krb5kdc[13333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626819, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.1@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:20:19.590656 13417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:19.580672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:57639 (local address 127.3.18.62:41665)
0928 20:20:19.580805 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:19.580807 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:19.580965 (+ 158us) server_negotiation.cc:407] Connection header received
0928 20:20:19.581606 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:19.581608 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:19.581620 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:19.581681 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:19.582685 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:19.583258 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:19.584353 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:19.584557 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:19.584646 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:19.587487 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:20:19.587489 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:20:19.587489 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:20:19.587498 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:20:19.589095 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:20:19.589628 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:20:19.589628 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:20:19.589629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:20:19.589681 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:20:19.589991 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:20:19.589991 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:20:19.589992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:20:19.590103 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:20:19.590154 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 20:20:19.590517 (+ 363us) server_negotiation.cc:299] Negotiation successful
0928 20:20:19.590587 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260928 20:20:19.590993 13571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:19.580589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:41665 (local address 127.3.18.1:57639)
0928 20:20:19.580819 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:19.580866 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 20:20:19.581441 (+ 575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:20:19.581758 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:20:19.581763 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:20:19.582366 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:20:19.582597 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:19.582604 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:19.583358 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:19.583359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:19.584255 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:19.584257 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:19.584658 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:19.584659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:19.584870 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:19.585526 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:20:19.585536 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:20:19.587318 (+ 1782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:20:19.589191 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:20:19.589201 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:20:19.589204 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:20:19.589547 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:20:19.589764 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:20:19.589765 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:20:19.589766 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:20:19.589923 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:20:19.590172 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:20:19.590179 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:20:19.590371 (+ 192us) client_negotiation.cc:769] Sending connection context
0928 20:20:19.590500 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 20:20:19.590697 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":83,"threads_started":1}
I20260928 20:20:19.591625 13569 heartbeater.cc:347] Connected to a master server at 127.3.18.62:41665
I20260928 20:20:19.591728 13569 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.3.18.2@KRBTEST.COM; defaulting to no policy
I20260928 20:20:19.591984 13569 heartbeater.cc:511] Master 127.3.18.62:41665 requested a full tablet report, sending...
I20260928 20:20:19.592649 13378 ts_manager.cc:194] Registered new tserver with Master: bbfa42bfe72742cea70d82f899336911 (127.3.18.1:39141)
I20260928 20:20:19.593374 13378 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.18.1@KRBTEST.COM'} at 127.3.18.1:57639
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab.
Entry for principal kudu/127.3.18.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab.
I20260928 20:20:19.611395 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.2:0
--local_ip_for_outbound_sockets=127.3.18.2
--webserver_interface=127.3.18.2
--webserver_port=0
--tserver_master_addrs=127.3.18.62:41665
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab
--principal=kudu/127.3.18.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:39535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:20:19.694233 13574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:20:19.694455 13574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:20:19.694479 13574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:20:19.695698 13574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:20:19.695749 13574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:20:19.695801 13574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.2
I20260928 20:20:19.697301 13574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:39535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab
--principal=kudu/127.3.18.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.18.2
--webserver_port=0
--tserver_master_addrs=127.3.18.62:41665
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13574
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.2
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:20:19.697508 13574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:20:19.697690 13574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:20:19.702957 13582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:20:19.702972 13579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:20:19.703042 13574 server_base.cc:1034] running on GCE node
W20260928 20:20:19.702984 13580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:20:19.703430 13574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:20:19.703825 13574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:20:19.704986 13574 hybrid_clock.cc:648] HybridClock initialized: now 1790626819704966 us; error 41 us; skew 500 ppm
Sep 28 20:20:19 dist-test-slave-b83v krb5kdc[13333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626819, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:20:19.707851 13574 init.cc:377] Logged in from keytab as kudu/127.3.18.2@KRBTEST.COM (short username kudu)
I20260928 20:20:19.708424 13574 webserver.cc:460] Webserver started at http://127.3.18.2:40553/ using document root <none> and password file <none>
I20260928 20:20:19.708686 13574 fs_manager.cc:359] Metadata directory not provided
I20260928 20:20:19.708735 13574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:20:19.708877 13574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:20:19.710026 13574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/minicluster-data/ts-1/data/instance:
uuid: "94cc42ad7f314dffab58a4f89620c868"
format_stamp: "Formatted at 2026-09-28 20:20:19 on dist-test-slave-b83v"
server_key: "2a0c97be3ea607e38cf2579c87c22b6b"
server_key_iv: "06259fc815ec162c08e47343a71ef06f"
server_key_version: "encryptionkey@0"
I20260928 20:20:19.710448 13574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/minicluster-data/ts-1/wal/instance:
uuid: "94cc42ad7f314dffab58a4f89620c868"
format_stamp: "Formatted at 2026-09-28 20:20:19 on dist-test-slave-b83v"
server_key: "2a0c97be3ea607e38cf2579c87c22b6b"
server_key_iv: "06259fc815ec162c08e47343a71ef06f"
server_key_version: "encryptionkey@0"
I20260928 20:20:19.712221 13574 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 20:20:19.713238 13589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:20:19.713449 13574 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:20:19.713563 13574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/minicluster-data/ts-1/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/minicluster-data/ts-1/wal
uuid: "94cc42ad7f314dffab58a4f89620c868"
format_stamp: "Formatted at 2026-09-28 20:20:19 on dist-test-slave-b83v"
server_key: "2a0c97be3ea607e38cf2579c87c22b6b"
server_key_iv: "06259fc815ec162c08e47343a71ef06f"
server_key_version: "encryptionkey@0"
I20260928 20:20:19.713644 13574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:20:19.725262 13574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:20:19.727702 13574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:20:19.727905 13574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:20:19.728166 13574 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:20:19.728492 13574 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:20:19.728559 13574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:20:19.728600 13574 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:20:19.728627 13574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:20:19.734807 13574 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.2:43409
I20260928 20:20:19.734911 13702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.2:43409 every 8 connection(s)
I20260928 20:20:19.735255 13574 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
I20260928 20:20:19.736994 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 13574
I20260928 20:20:19.737102 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/minicluster-data/ts-1/wal/instance
I20260928 20:20:19.737344 3144 external_mini_cluster.cc:1428] Setting key 0026bd94148c2dc9a6d87db6ade80141
Sep 28 20:20:19 dist-test-slave-b83v krb5kdc[13333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626819, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.2@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.18.3@KRBTEST.COM; defaulting to no policy
I20260928 20:20:19.746057 13417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:19.736046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:59129 (local address 127.3.18.62:41665)
0928 20:20:19.736105 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:19.736107 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:19.736304 (+ 197us) server_negotiation.cc:407] Connection header received
0928 20:20:19.736849 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:19.736851 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:19.736864 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:19.736911 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:19.738205 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:19.739000 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:19.740479 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:19.740659 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:19.740714 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:19.743124 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:20:19.743126 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:20:19.743126 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:20:19.743135 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:20:19.744607 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:20:19.745057 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:20:19.745058 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:20:19.745059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:20:19.745120 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:20:19.745388 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:20:19.745389 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:20:19.745390 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:20:19.745536 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:20:19.745606 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 20:20:19.745885 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 20:20:19.745964 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 20:20:19.746259 13705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:19.736037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:41665 (local address 127.3.18.2:59129)
0928 20:20:19.736188 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:19.736224 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 20:20:19.736724 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:20:19.737003 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:20:19.737006 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:20:19.737793 (+ 787us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:20:19.738081 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:19.738088 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:19.739120 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:19.739121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:19.740395 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:19.740399 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:19.740743 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:19.740744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:19.740936 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:19.741464 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:20:19.741474 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:20:19.743029 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:20:19.744726 (+ 1697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:20:19.744732 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:20:19.744735 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:20:19.744965 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:20:19.745193 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:20:19.745194 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:20:19.745194 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:20:19.745314 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:20:19.745608 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:20:19.745615 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:20:19.745755 (+ 140us) client_negotiation.cc:769] Sending connection context
0928 20:20:19.745862 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 20:20:19.746020 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":73,"threads_started":1}
I20260928 20:20:19.746742 13703 heartbeater.cc:347] Connected to a master server at 127.3.18.62:41665
I20260928 20:20:19.746857 13703 heartbeater.cc:464] Registering TS with master...
I20260928 20:20:19.747037 13703 heartbeater.cc:511] Master 127.3.18.62:41665 requested a full tablet report, sending...
I20260928 20:20:19.747496 13378 ts_manager.cc:194] Registered new tserver with Master: 94cc42ad7f314dffab58a4f89620c868 (127.3.18.2:43409)
I20260928 20:20:19.747947 13378 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.18.2@KRBTEST.COM'} at 127.3.18.2:59129
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab.
Entry for principal kudu/127.3.18.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab.
I20260928 20:20:19.767946 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.3:0
--local_ip_for_outbound_sockets=127.3.18.3
--webserver_interface=127.3.18.3
--webserver_port=0
--tserver_master_addrs=127.3.18.62:41665
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab
--principal=kudu/127.3.18.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:39535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:20:19.852523 13708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:20:19.852799 13708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:20:19.852854 13708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:20:19.854188 13708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:20:19.854277 13708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:20:19.854333 13708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.3
I20260928 20:20:19.856316 13708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:39535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab
--principal=kudu/127.3.18.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.18.3
--webserver_port=0
--tserver_master_addrs=127.3.18.62:41665
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.3
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:20:19.856637 13708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:20:19.856935 13708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:20:19.862573 13716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:20:19.862571 13714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:20:19.862565 13713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:20:19.863057 13708 server_base.cc:1034] running on GCE node
I20260928 20:20:19.863389 13708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:20:19.863695 13708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:20:19.864866 13708 hybrid_clock.cc:648] HybridClock initialized: now 1790626819864807 us; error 68 us; skew 500 ppm
Sep 28 20:20:19 dist-test-slave-b83v krb5kdc[13333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626819, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:20:19.868221 13708 init.cc:377] Logged in from keytab as kudu/127.3.18.3@KRBTEST.COM (short username kudu)
I20260928 20:20:19.868757 13708 webserver.cc:460] Webserver started at http://127.3.18.3:38535/ using document root <none> and password file <none>
I20260928 20:20:19.869043 13708 fs_manager.cc:359] Metadata directory not provided
I20260928 20:20:19.869092 13708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:20:19.869246 13708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:20:19.870374 13708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/minicluster-data/ts-2/data/instance:
uuid: "228f345b6bab4f8789f4c6d52b3a2e3f"
format_stamp: "Formatted at 2026-09-28 20:20:19 on dist-test-slave-b83v"
server_key: "1975c77e3ecea8372ad18ea0b69edf40"
server_key_iv: "56765317c849054e1df4c782881b19b3"
server_key_version: "encryptionkey@0"
I20260928 20:20:19.870749 13708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/minicluster-data/ts-2/wal/instance:
uuid: "228f345b6bab4f8789f4c6d52b3a2e3f"
format_stamp: "Formatted at 2026-09-28 20:20:19 on dist-test-slave-b83v"
server_key: "1975c77e3ecea8372ad18ea0b69edf40"
server_key_iv: "56765317c849054e1df4c782881b19b3"
server_key_version: "encryptionkey@0"
I20260928 20:20:19.873035 13708 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260928 20:20:19.874100 13723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:20:19.874451 13708 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:20:19.874581 13708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/minicluster-data/ts-2/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/minicluster-data/ts-2/wal
uuid: "228f345b6bab4f8789f4c6d52b3a2e3f"
format_stamp: "Formatted at 2026-09-28 20:20:19 on dist-test-slave-b83v"
server_key: "1975c77e3ecea8372ad18ea0b69edf40"
server_key_iv: "56765317c849054e1df4c782881b19b3"
server_key_version: "encryptionkey@0"
I20260928 20:20:19.874675 13708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:20:19.900100 13708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:20:19.902666 13708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:20:19.902827 13708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:20:19.903074 13708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:20:19.903410 13708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:20:19.903466 13708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:20:19.903508 13708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:20:19.903535 13708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:20:19.909619 13708 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.3:43821
I20260928 20:20:19.909657 13836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.3:43821 every 8 connection(s)
I20260928 20:20:19.910106 13708 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
I20260928 20:20:19.913440 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 13708
I20260928 20:20:19.913560 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/minicluster-data/ts-2/wal/instance
I20260928 20:20:19.913769 3144 external_mini_cluster.cc:1428] Setting key 335fed5414e4821d00fba48a9cb4f56a
Sep 28 20:20:19 dist-test-slave-b83v krb5kdc[13333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626819, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.3@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:20:19.920706 13417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:19.911044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:35387 (local address 127.3.18.62:41665)
0928 20:20:19.911111 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:19.911113 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:19.911353 (+ 240us) server_negotiation.cc:407] Connection header received
0928 20:20:19.911960 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:19.911961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:19.911973 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:19.912029 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:19.913006 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:19.913522 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:19.914932 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:19.915198 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:19.915266 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:19.917624 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:20:19.917625 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:20:19.917626 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:20:19.917633 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:20:19.919288 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:20:19.919735 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:20:19.919736 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:20:19.919737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:20:19.919856 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:20:19.920093 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:20:19.920094 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:20:19.920094 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:20:19.920209 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:20:19.920293 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 20:20:19.920566 (+ 273us) server_negotiation.cc:299] Negotiation successful
0928 20:20:19.920635 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 20:20:19.920939 13839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:19.911045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:41665 (local address 127.3.18.3:35387)
0928 20:20:19.911223 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:19.911261 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 20:20:19.911850 (+ 589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:20:19.912108 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:20:19.912112 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:20:19.912679 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:20:19.912893 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:19.912899 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:19.913663 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:19.913665 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:19.914811 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:19.914815 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:19.915310 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:19.915311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:19.915424 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:19.916080 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:20:19.916089 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:20:19.917531 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:20:19.919385 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:20:19.919391 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:20:19.919393 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:20:19.919648 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:20:19.919917 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:20:19.919917 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:20:19.919918 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:20:19.920029 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:20:19.920288 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:20:19.920294 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:20:19.920424 (+ 130us) client_negotiation.cc:769] Sending connection context
0928 20:20:19.920522 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 20:20:19.920649 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":96,"threads_started":1}
I20260928 20:20:19.921545 13837 heartbeater.cc:347] Connected to a master server at 127.3.18.62:41665
I20260928 20:20:19.921648 13837 heartbeater.cc:464] Registering TS with master...
I20260928 20:20:19.921861 13837 heartbeater.cc:511] Master 127.3.18.62:41665 requested a full tablet report, sending...
I20260928 20:20:19.922422 13378 ts_manager.cc:194] Registered new tserver with Master: 228f345b6bab4f8789f4c6d52b3a2e3f (127.3.18.3:43821)
I20260928 20:20:19.922888 13378 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.18.3@KRBTEST.COM'} at 127.3.18.3:35387
I20260928 20:20:19.926287 3144 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 20:20:19.934788 13417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:19.927909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45830 (local address 127.3.18.62:41665)
0928 20:20:19.927966 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:19.927968 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:19.928016 (+ 48us) server_negotiation.cc:407] Connection header received
0928 20:20:19.928091 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:19.928091 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:19.928101 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:19.928146 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:20:19.928895 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:19.929448 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:19.930278 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:19.930498 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:19.930666 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:19.932235 (+ 1569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:20:19.932237 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:20:19.932237 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:20:19.932245 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:20:19.933481 (+ 1236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:20:19.933853 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:20:19.933854 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:20:19.933854 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:20:19.933919 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:20:19.934139 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:20:19.934140 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:20:19.934140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:20:19.934313 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:20:19.934426 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 20:20:19.934565 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 20:20:19.934659 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 20:20:19.936887 13378 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45830:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 20:20:19.938705 13378 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 20:20:19.945797 13853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:19.941893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39196 (local address 127.3.18.3:43821)
0928 20:20:19.942135 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:19.942141 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:19.942152 (+ 11us) server_negotiation.cc:407] Connection header received
0928 20:20:19.942198 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:19.942200 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:19.942217 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:19.942288 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:20:19.942441 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:19.943129 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:19.944622 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:19.945399 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:19.945482 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:19.945544 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 20:20:19.945688 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 20:20:19.945726 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":87,"threads_started":1}
I20260928 20:20:19.945816 13850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:19.941525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.3:43821 (local address 127.0.0.1:39196)
0928 20:20:19.942072 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:19.942084 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:20:19.942129 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:20:19.942295 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:20:19.942296 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:20:19.942298 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:20:19.942363 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:19.942365 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:19.943240 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:19.943241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:19.944523 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:19.944526 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:19.945497 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:19.945498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:19.945580 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:19.945594 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 20:20:19.945677 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 20:20:19.945735 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"mutex_wait_us":76,"thread_start_us":92,"threads_started":1}
I20260928 20:20:19.946228 13848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:19.941516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.2:43409 (local address 127.0.0.1:56312)
0928 20:20:19.941697 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:19.941738 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 20:20:19.941809 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:20:19.942176 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:20:19.942179 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:20:19.942181 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:20:19.942253 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:19.942254 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:19.943200 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:19.943201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:19.944625 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:19.944629 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:19.946001 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:19.946002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:19.946062 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:19.946065 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 20:20:19.946112 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 20:20:19.946158 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":35,"thread_start_us":94,"threads_started":1}
I20260928 20:20:19.946610 13849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:19.941520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.1:39141 (local address 127.0.0.1:33464)
0928 20:20:19.941995 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:19.942006 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:20:19.942052 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:20:19.943466 (+ 1414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:20:19.943467 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:20:19.943469 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:20:19.943520 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:19.943521 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:19.944774 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:19.944775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:19.945486 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:19.945488 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:19.946392 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:19.946392 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:19.946450 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:19.946454 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:20:19.946512 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 20:20:19.946546 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"mutex_wait_us":2,"thread_start_us":65,"threads_started":1}
I20260928 20:20:19.946638 13852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:19.941822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33464 (local address 127.3.18.1:39141)
0928 20:20:19.943293 (+ 1471us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:19.943297 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:19.943307 (+ 10us) server_negotiation.cc:407] Connection header received
0928 20:20:19.943350 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:19.943352 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:19.943367 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:19.943446 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:20:19.943590 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:19.944676 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:19.945580 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:19.946316 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:19.946390 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:19.946447 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 20:20:19.946525 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 20:20:19.946560 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1432,"thread_start_us":92,"threads_started":1}
I20260928 20:20:19.946636 13771 tablet_service.cc:1467] Processing CreateTablet for tablet 9255e8e7316a476a9ebafb666b0f5902 (DEFAULT_TABLE table=test-table [id=28458b5e9bb0402fac6b9d6452675318]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:20:19.946635 13851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:19.941704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56312 (local address 127.3.18.2:43409)
0928 20:20:19.941858 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:19.941864 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:19.941876 (+ 12us) server_negotiation.cc:407] Connection header received
0928 20:20:19.941962 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:19.941964 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:19.941982 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:19.942074 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:20:19.942338 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:19.943091 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:19.944719 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:19.945407 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:19.945482 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:19.945544 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 20:20:19.946501 (+ 957us) server_negotiation.cc:299] Negotiation successful
0928 20:20:19.946555 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":85,"threads_started":1}
I20260928 20:20:19.947053 13771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9255e8e7316a476a9ebafb666b0f5902. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:20:19.947587 13637 tablet_service.cc:1467] Processing CreateTablet for tablet 9255e8e7316a476a9ebafb666b0f5902 (DEFAULT_TABLE table=test-table [id=28458b5e9bb0402fac6b9d6452675318]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:20:19.947975 13637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9255e8e7316a476a9ebafb666b0f5902. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:20:19.950845 13854 tablet_bootstrap.cc:492] T 9255e8e7316a476a9ebafb666b0f5902 P 228f345b6bab4f8789f4c6d52b3a2e3f: Bootstrap starting.
I20260928 20:20:19.951665 13855 tablet_bootstrap.cc:492] T 9255e8e7316a476a9ebafb666b0f5902 P 94cc42ad7f314dffab58a4f89620c868: Bootstrap starting.
I20260928 20:20:19.951833 13854 tablet_bootstrap.cc:654] T 9255e8e7316a476a9ebafb666b0f5902 P 228f345b6bab4f8789f4c6d52b3a2e3f: Neither blocks nor log segments found. Creating new log.
I20260928 20:20:19.951700 13503 tablet_service.cc:1467] Processing CreateTablet for tablet 9255e8e7316a476a9ebafb666b0f5902 (DEFAULT_TABLE table=test-table [id=28458b5e9bb0402fac6b9d6452675318]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:20:19.952128 13503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9255e8e7316a476a9ebafb666b0f5902. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:20:19.952371 13855 tablet_bootstrap.cc:654] T 9255e8e7316a476a9ebafb666b0f5902 P 94cc42ad7f314dffab58a4f89620c868: Neither blocks nor log segments found. Creating new log.
I20260928 20:20:19.952455 13854 log.cc:824] T 9255e8e7316a476a9ebafb666b0f5902 P 228f345b6bab4f8789f4c6d52b3a2e3f: Log is configured to *not* fsync() on all Append() calls
I20260928 20:20:19.953442 13854 tablet_bootstrap.cc:492] T 9255e8e7316a476a9ebafb666b0f5902 P 228f345b6bab4f8789f4c6d52b3a2e3f: No bootstrap required, opened a new log
I20260928 20:20:19.953532 13854 ts_tablet_manager.cc:1397] T 9255e8e7316a476a9ebafb666b0f5902 P 228f345b6bab4f8789f4c6d52b3a2e3f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 20:20:19.954227 13855 log.cc:824] T 9255e8e7316a476a9ebafb666b0f5902 P 94cc42ad7f314dffab58a4f89620c868: Log is configured to *not* fsync() on all Append() calls
I20260928 20:20:19.954969 13855 tablet_bootstrap.cc:492] T 9255e8e7316a476a9ebafb666b0f5902 P 94cc42ad7f314dffab58a4f89620c868: No bootstrap required, opened a new log
I20260928 20:20:19.955060 13855 ts_tablet_manager.cc:1397] T 9255e8e7316a476a9ebafb666b0f5902 P 94cc42ad7f314dffab58a4f89620c868: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 20:20:19.955929 13854 raft_consensus.cc:348] T 9255e8e7316a476a9ebafb666b0f5902 P 228f345b6bab4f8789f4c6d52b3a2e3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbfa42bfe72742cea70d82f899336911" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 39141 } } peers { permanent_uuid: "94cc42ad7f314dffab58a4f89620c868" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 43409 } } peers { permanent_uuid: "228f345b6bab4f8789f4c6d52b3a2e3f" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 43821 } }
I20260928 20:20:19.956146 13854 raft_consensus.cc:374] T 9255e8e7316a476a9ebafb666b0f5902 P 228f345b6bab4f8789f4c6d52b3a2e3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:20:19.956202 13854 raft_consensus.cc:729] T 9255e8e7316a476a9ebafb666b0f5902 P 228f345b6bab4f8789f4c6d52b3a2e3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 228f345b6bab4f8789f4c6d52b3a2e3f, State: Initialized, Role: FOLLOWER
I20260928 20:20:19.956329 13854 consensus_queue.cc:260] T 9255e8e7316a476a9ebafb666b0f5902 P 228f345b6bab4f8789f4c6d52b3a2e3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbfa42bfe72742cea70d82f899336911" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 39141 } } peers { permanent_uuid: "94cc42ad7f314dffab58a4f89620c868" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 43409 } } peers { permanent_uuid: "228f345b6bab4f8789f4c6d52b3a2e3f" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 43821 } }
I20260928 20:20:19.956748 13837 heartbeater.cc:503] Master 127.3.18.62:41665 was elected leader, sending a full tablet report...
I20260928 20:20:19.957048 13854 ts_tablet_manager.cc:1428] T 9255e8e7316a476a9ebafb666b0f5902 P 228f345b6bab4f8789f4c6d52b3a2e3f: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 20:20:19.957015 13855 raft_consensus.cc:348] T 9255e8e7316a476a9ebafb666b0f5902 P 94cc42ad7f314dffab58a4f89620c868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbfa42bfe72742cea70d82f899336911" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 39141 } } peers { permanent_uuid: "94cc42ad7f314dffab58a4f89620c868" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 43409 } } peers { permanent_uuid: "228f345b6bab4f8789f4c6d52b3a2e3f" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 43821 } }
I20260928 20:20:19.957175 13855 raft_consensus.cc:374] T 9255e8e7316a476a9ebafb666b0f5902 P 94cc42ad7f314dffab58a4f89620c868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:20:19.957214 13855 raft_consensus.cc:729] T 9255e8e7316a476a9ebafb666b0f5902 P 94cc42ad7f314dffab58a4f89620c868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94cc42ad7f314dffab58a4f89620c868, State: Initialized, Role: FOLLOWER
I20260928 20:20:19.957357 13855 consensus_queue.cc:260] T 9255e8e7316a476a9ebafb666b0f5902 P 94cc42ad7f314dffab58a4f89620c868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbfa42bfe72742cea70d82f899336911" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 39141 } } peers { permanent_uuid: "94cc42ad7f314dffab58a4f89620c868" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 43409 } } peers { permanent_uuid: "228f345b6bab4f8789f4c6d52b3a2e3f" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 43821 } }
I20260928 20:20:19.957635 13855 ts_tablet_manager.cc:1428] T 9255e8e7316a476a9ebafb666b0f5902 P 94cc42ad7f314dffab58a4f89620c868: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 20:20:19.957741 13703 heartbeater.cc:503] Master 127.3.18.62:41665 was elected leader, sending a full tablet report...
I20260928 20:20:19.957769 13858 tablet_bootstrap.cc:492] T 9255e8e7316a476a9ebafb666b0f5902 P bbfa42bfe72742cea70d82f899336911: Bootstrap starting.
I20260928 20:20:19.958694 13858 tablet_bootstrap.cc:654] T 9255e8e7316a476a9ebafb666b0f5902 P bbfa42bfe72742cea70d82f899336911: Neither blocks nor log segments found. Creating new log.
I20260928 20:20:19.959139 13858 log.cc:824] T 9255e8e7316a476a9ebafb666b0f5902 P bbfa42bfe72742cea70d82f899336911: Log is configured to *not* fsync() on all Append() calls
I20260928 20:20:19.959952 13858 tablet_bootstrap.cc:492] T 9255e8e7316a476a9ebafb666b0f5902 P bbfa42bfe72742cea70d82f899336911: No bootstrap required, opened a new log
I20260928 20:20:19.960009 13858 ts_tablet_manager.cc:1397] T 9255e8e7316a476a9ebafb666b0f5902 P bbfa42bfe72742cea70d82f899336911: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 20:20:19.961334 13858 raft_consensus.cc:348] T 9255e8e7316a476a9ebafb666b0f5902 P bbfa42bfe72742cea70d82f899336911 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbfa42bfe72742cea70d82f899336911" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 39141 } } peers { permanent_uuid: "94cc42ad7f314dffab58a4f89620c868" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 43409 } } peers { permanent_uuid: "228f345b6bab4f8789f4c6d52b3a2e3f" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 43821 } }
I20260928 20:20:19.961463 13858 raft_consensus.cc:374] T 9255e8e7316a476a9ebafb666b0f5902 P bbfa42bfe72742cea70d82f899336911 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:20:19.961484 13858 raft_consensus.cc:729] T 9255e8e7316a476a9ebafb666b0f5902 P bbfa42bfe72742cea70d82f899336911 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbfa42bfe72742cea70d82f899336911, State: Initialized, Role: FOLLOWER
I20260928 20:20:19.961599 13858 consensus_queue.cc:260] T 9255e8e7316a476a9ebafb666b0f5902 P bbfa42bfe72742cea70d82f899336911 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbfa42bfe72742cea70d82f899336911" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 39141 } } peers { permanent_uuid: "94cc42ad7f314dffab58a4f89620c868" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 43409 } } peers { permanent_uuid: "228f345b6bab4f8789f4c6d52b3a2e3f" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 43821 } }
I20260928 20:20:19.961915 13858 ts_tablet_manager.cc:1428] T 9255e8e7316a476a9ebafb666b0f5902 P bbfa42bfe72742cea70d82f899336911: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 20:20:19.961938 13569 heartbeater.cc:503] Master 127.3.18.62:41665 was elected leader, sending a full tablet report...
W20260928 20:20:19.985944 13704 tablet.cc:2369] T 9255e8e7316a476a9ebafb666b0f5902 P 94cc42ad7f314dffab58a4f89620c868: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 20:20:20.080591 13570 tablet.cc:2369] T 9255e8e7316a476a9ebafb666b0f5902 P bbfa42bfe72742cea70d82f899336911: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 20:20:20.155675 13859 raft_consensus.cc:482] T 9255e8e7316a476a9ebafb666b0f5902 P 228f345b6bab4f8789f4c6d52b3a2e3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:20:20.155921 13859 raft_consensus.cc:504] T 9255e8e7316a476a9ebafb666b0f5902 P 228f345b6bab4f8789f4c6d52b3a2e3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbfa42bfe72742cea70d82f899336911" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 39141 } } peers { permanent_uuid: "94cc42ad7f314dffab58a4f89620c868" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 43409 } } peers { permanent_uuid: "228f345b6bab4f8789f4c6d52b3a2e3f" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 43821 } }
I20260928 20:20:20.156466 13859 leader_election.cc:288] T 9255e8e7316a476a9ebafb666b0f5902 P 228f345b6bab4f8789f4c6d52b3a2e3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbfa42bfe72742cea70d82f899336911 (127.3.18.1:39141), 94cc42ad7f314dffab58a4f89620c868 (127.3.18.2:43409)
W20260928 20:20:20.160660 13838 tablet.cc:2369] T 9255e8e7316a476a9ebafb666b0f5902 P 228f345b6bab4f8789f4c6d52b3a2e3f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 20:20:20.160899 13851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:20.156867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:41809 (local address 127.3.18.2:43409)
0928 20:20:20.156913 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:20.156915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:20.157097 (+ 182us) server_negotiation.cc:407] Connection header received
0928 20:20:20.157134 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:20.157135 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:20.157146 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:20.157251 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:20:20.157416 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:20.158093 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:20.159727 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:20.160413 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:20.160499 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:20.160511 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 20:20:20.160782 (+ 271us) server_negotiation.cc:299] Negotiation successful
0928 20:20:20.160833 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 20:20:20.160907 13839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:20.156712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.1:39141 (local address 127.3.18.3:59077)
0928 20:20:20.156778 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:20.156793 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 20:20:20.156849 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:20:20.157054 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:20:20.157055 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:20:20.157057 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:20:20.157138 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:20.157141 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:20.158164 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:20.158166 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:20.159565 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:20.159569 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:20.160575 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:20.160576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:20.160678 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:20.160681 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 20:20:20.160747 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 20:20:20.160803 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":74}
I20260928 20:20:20.160918 13863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:20.156716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.2:43409 (local address 127.3.18.3:41809)
0928 20:20:20.157008 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:20.157018 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 20:20:20.157073 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:20:20.157277 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:20:20.157278 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:20:20.157279 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:20:20.157338 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:20.157339 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:20.158194 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:20.158194 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:20.159565 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:20:20.159569 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:20:20.160575 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:20:20.160576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:20:20.160671 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:20.160676 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 20:20:20.160758 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 20:20:20.160803 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":153,"thread_start_us":120,"threads_started":1}
I20260928 20:20:20.161203 13852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:20.156835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:59077 (local address 127.3.18.1:39141)
0928 20:20:20.156883 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:20.156885 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:20.156897 (+ 12us) server_negotiation.cc:407] Connection header received
0928 20:20:20.156952 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:20.156953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:20.156964 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:20.157037 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:20:20.157246 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:20.158015 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:20.159727 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:20.160397 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:20.160499 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:20.160511 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 20:20:20.161053 (+ 542us) server_negotiation.cc:299] Negotiation successful
0928 20:20:20.161119 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 20:20:20.161538 13523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9255e8e7316a476a9ebafb666b0f5902" candidate_uuid: "228f345b6bab4f8789f4c6d52b3a2e3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbfa42bfe72742cea70d82f899336911" is_pre_election: true
I20260928 20:20:20.161566 13657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9255e8e7316a476a9ebafb666b0f5902" candidate_uuid: "228f345b6bab4f8789f4c6d52b3a2e3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94cc42ad7f314dffab58a4f89620c868" is_pre_election: true
I20260928 20:20:20.161746 13523 raft_consensus.cc:2454] T 9255e8e7316a476a9ebafb666b0f5902 P bbfa42bfe72742cea70d82f899336911 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 228f345b6bab4f8789f4c6d52b3a2e3f in term 0.
I20260928 20:20:20.161762 13657 raft_consensus.cc:2454] T 9255e8e7316a476a9ebafb666b0f5902 P 94cc42ad7f314dffab58a4f89620c868 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 228f345b6bab4f8789f4c6d52b3a2e3f in term 0.
I20260928 20:20:20.162036 13726 leader_election.cc:302] T 9255e8e7316a476a9ebafb666b0f5902 P 228f345b6bab4f8789f4c6d52b3a2e3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 228f345b6bab4f8789f4c6d52b3a2e3f, bbfa42bfe72742cea70d82f899336911; no voters:
I20260928 20:20:20.162274 13859 raft_consensus.cc:2781] T 9255e8e7316a476a9ebafb666b0f5902 P 228f345b6bab4f8789f4c6d52b3a2e3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 20:20:20.162364 13859 raft_consensus.cc:482] T 9255e8e7316a476a9ebafb666b0f5902 P 228f345b6bab4f8789f4c6d52b3a2e3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 20:20:20.162402 13859 raft_consensus.cc:3037] T 9255e8e7316a476a9ebafb666b0f5902 P 228f345b6bab4f8789f4c6d52b3a2e3f [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:20:20.163203 13859 raft_consensus.cc:504] T 9255e8e7316a476a9ebafb666b0f5902 P 228f345b6bab4f8789f4c6d52b3a2e3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbfa42bfe72742cea70d82f899336911" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 39141 } } peers { permanent_uuid: "94cc42ad7f314dffab58a4f89620c868" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 43409 } } peers { permanent_uuid: "228f345b6bab4f8789f4c6d52b3a2e3f" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 43821 } }
I20260928 20:20:20.163403 13859 leader_election.cc:288] T 9255e8e7316a476a9ebafb666b0f5902 P 228f345b6bab4f8789f4c6d52b3a2e3f [CANDIDATE]: Term 1 election: Requested vote from peers bbfa42bfe72742cea70d82f899336911 (127.3.18.1:39141), 94cc42ad7f314dffab58a4f89620c868 (127.3.18.2:43409)
I20260928 20:20:20.163695 13657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9255e8e7316a476a9ebafb666b0f5902" candidate_uuid: "228f345b6bab4f8789f4c6d52b3a2e3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94cc42ad7f314dffab58a4f89620c868"
I20260928 20:20:20.163695 13523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9255e8e7316a476a9ebafb666b0f5902" candidate_uuid: "228f345b6bab4f8789f4c6d52b3a2e3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbfa42bfe72742cea70d82f899336911"
I20260928 20:20:20.163842 13657 raft_consensus.cc:3037] T 9255e8e7316a476a9ebafb666b0f5902 P 94cc42ad7f314dffab58a4f89620c868 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:20:20.163933 13523 raft_consensus.cc:3037] T 9255e8e7316a476a9ebafb666b0f5902 P bbfa42bfe72742cea70d82f899336911 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:20:20.164919 13523 raft_consensus.cc:2454] T 9255e8e7316a476a9ebafb666b0f5902 P bbfa42bfe72742cea70d82f899336911 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 228f345b6bab4f8789f4c6d52b3a2e3f in term 1.
I20260928 20:20:20.164919 13657 raft_consensus.cc:2454] T 9255e8e7316a476a9ebafb666b0f5902 P 94cc42ad7f314dffab58a4f89620c868 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 228f345b6bab4f8789f4c6d52b3a2e3f in term 1.
I20260928 20:20:20.165236 13725 leader_election.cc:302] T 9255e8e7316a476a9ebafb666b0f5902 P 228f345b6bab4f8789f4c6d52b3a2e3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 228f345b6bab4f8789f4c6d52b3a2e3f, 94cc42ad7f314dffab58a4f89620c868; no voters:
I20260928 20:20:20.165361 13859 raft_consensus.cc:2781] T 9255e8e7316a476a9ebafb666b0f5902 P 228f345b6bab4f8789f4c6d52b3a2e3f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:20:20.165552 13859 raft_consensus.cc:686] T 9255e8e7316a476a9ebafb666b0f5902 P 228f345b6bab4f8789f4c6d52b3a2e3f [term 1 LEADER]: Becoming Leader. State: Replica: 228f345b6bab4f8789f4c6d52b3a2e3f, State: Running, Role: LEADER
I20260928 20:20:20.165691 13859 consensus_queue.cc:237] T 9255e8e7316a476a9ebafb666b0f5902 P 228f345b6bab4f8789f4c6d52b3a2e3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbfa42bfe72742cea70d82f899336911" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 39141 } } peers { permanent_uuid: "94cc42ad7f314dffab58a4f89620c868" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 43409 } } peers { permanent_uuid: "228f345b6bab4f8789f4c6d52b3a2e3f" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 43821 } }
I20260928 20:20:20.166893 13377 catalog_manager.cc:5495] T 9255e8e7316a476a9ebafb666b0f5902 P 228f345b6bab4f8789f4c6d52b3a2e3f reported cstate change: term changed from 0 to 1, leader changed from <none> to 228f345b6bab4f8789f4c6d52b3a2e3f (127.3.18.3). New cstate: current_term: 1 leader_uuid: "228f345b6bab4f8789f4c6d52b3a2e3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbfa42bfe72742cea70d82f899336911" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 39141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94cc42ad7f314dffab58a4f89620c868" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 43409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "228f345b6bab4f8789f4c6d52b3a2e3f" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 43821 } health_report { overall_health: HEALTHY } } }
I20260928 20:20:20.180581 13853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:20.177496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39210 (local address 127.3.18.3:43821)
0928 20:20:20.177581 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0928 20:20:20.177584 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:20:20.177594 (+ 10us) server_negotiation.cc:407] Connection header received
0928 20:20:20.177673 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:20:20.177674 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:20:20.177697 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:20:20.177788 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:20:20.177969 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:20.178493 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:20.179459 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:20:20.179665 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:20:20.179882 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:20:20.180101 (+ 219us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:20:20.180256 (+ 155us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:20:20.180337 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 20:20:20.180427 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 20:20:20.180451 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 20:20:20.182868 13523 raft_consensus.cc:1261] T 9255e8e7316a476a9ebafb666b0f5902 P bbfa42bfe72742cea70d82f899336911 [term 1 FOLLOWER]: Refusing update from remote peer 228f345b6bab4f8789f4c6d52b3a2e3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:20:20.182860 13657 raft_consensus.cc:1261] T 9255e8e7316a476a9ebafb666b0f5902 P 94cc42ad7f314dffab58a4f89620c868 [term 1 FOLLOWER]: Refusing update from remote peer 228f345b6bab4f8789f4c6d52b3a2e3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:20:20.183310 13864 consensus_queue.cc:1035] T 9255e8e7316a476a9ebafb666b0f5902 P 228f345b6bab4f8789f4c6d52b3a2e3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "94cc42ad7f314dffab58a4f89620c868" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 43409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:20:20.183482 13859 consensus_queue.cc:1035] T 9255e8e7316a476a9ebafb666b0f5902 P 228f345b6bab4f8789f4c6d52b3a2e3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbfa42bfe72742cea70d82f899336911" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 39141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:20:20.189436 13378 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:20:20.189599 13378 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:20:20.190734 13378 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8455ccb20cf14d4cabd1a394435e7c34: Sending DeleteTablet for 3 replicas of tablet 9255e8e7316a476a9ebafb666b0f5902
I20260928 20:20:20.191203 13503 tablet_service.cc:1514] Processing DeleteTablet for tablet 9255e8e7316a476a9ebafb666b0f5902 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:20:20 UTC) from {username='kudu', principal='kudu/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:33464
I20260928 20:20:20.191211 13771 tablet_service.cc:1514] Processing DeleteTablet for tablet 9255e8e7316a476a9ebafb666b0f5902 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:20:20 UTC) from {username='kudu', principal='kudu/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:39196
I20260928 20:20:20.191569 13877 tablet_replica.cc:330] T 9255e8e7316a476a9ebafb666b0f5902 P 228f345b6bab4f8789f4c6d52b3a2e3f: stopping tablet replica
I20260928 20:20:20.191569 13878 tablet_replica.cc:330] T 9255e8e7316a476a9ebafb666b0f5902 P bbfa42bfe72742cea70d82f899336911: stopping tablet replica
I20260928 20:20:20.191691 13878 raft_consensus.cc:2229] T 9255e8e7316a476a9ebafb666b0f5902 P bbfa42bfe72742cea70d82f899336911 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:20:20.191691 13877 raft_consensus.cc:2229] T 9255e8e7316a476a9ebafb666b0f5902 P 228f345b6bab4f8789f4c6d52b3a2e3f [term 1 LEADER]: Raft consensus shutting down.
I20260928 20:20:20.191718 13637 tablet_service.cc:1514] Processing DeleteTablet for tablet 9255e8e7316a476a9ebafb666b0f5902 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:20:20 UTC) from {username='kudu', principal='kudu/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:56312
I20260928 20:20:20.191866 13878 raft_consensus.cc:2258] T 9255e8e7316a476a9ebafb666b0f5902 P bbfa42bfe72742cea70d82f899336911 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:20:20.191946 13877 raft_consensus.cc:2258] T 9255e8e7316a476a9ebafb666b0f5902 P 228f345b6bab4f8789f4c6d52b3a2e3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:20:20.192257 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 13440
I20260928 20:20:20.194265 13877 ts_tablet_manager.cc:1905] T 9255e8e7316a476a9ebafb666b0f5902 P 228f345b6bab4f8789f4c6d52b3a2e3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:20:20.196213 13877 ts_tablet_manager.cc:1918] T 9255e8e7316a476a9ebafb666b0f5902 P 228f345b6bab4f8789f4c6d52b3a2e3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:20:20.196400 13877 log.cc:1196] T 9255e8e7316a476a9ebafb666b0f5902 P 228f345b6bab4f8789f4c6d52b3a2e3f: Deleting WAL directory at /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/minicluster-data/ts-2/wal/wals/9255e8e7316a476a9ebafb666b0f5902
I20260928 20:20:20.196841 13877 ts_tablet_manager.cc:1939] T 9255e8e7316a476a9ebafb666b0f5902 P 228f345b6bab4f8789f4c6d52b3a2e3f: Deleting consensus metadata
I20260928 20:20:20.197674 13362 catalog_manager.cc:4841] TS 228f345b6bab4f8789f4c6d52b3a2e3f (127.3.18.3:43821): tablet 9255e8e7316a476a9ebafb666b0f5902 (table test-table [id=28458b5e9bb0402fac6b9d6452675318]) successfully deleted
W20260928 20:20:20.198853 13364 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.18.1:39141 (error 108)
I20260928 20:20:20.199008 13879 tablet_replica.cc:330] T 9255e8e7316a476a9ebafb666b0f5902 P 94cc42ad7f314dffab58a4f89620c868: stopping tablet replica
I20260928 20:20:20.199172 13879 raft_consensus.cc:2229] T 9255e8e7316a476a9ebafb666b0f5902 P 94cc42ad7f314dffab58a4f89620c868 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:20:20.199326 13879 raft_consensus.cc:2258] T 9255e8e7316a476a9ebafb666b0f5902 P 94cc42ad7f314dffab58a4f89620c868 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:20:20.199651 13849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:20:20.199487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.1:39141 (local address 127.0.0.1:33468)
0928 20:20:20.199553 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 20:20:20.199609 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.1:39141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 20:20:20.199848 13364 catalog_manager.cc:4584] TS bbfa42bfe72742cea70d82f899336911 (127.3.18.1:39141): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9255e8e7316a476a9ebafb666b0f5902: Network error: Client connection negotiation failed: client connection to 127.3.18.1:39141: connect: Connection refused (error 111)
I20260928 20:20:20.200210 13879 ts_tablet_manager.cc:1905] T 9255e8e7316a476a9ebafb666b0f5902 P 94cc42ad7f314dffab58a4f89620c868: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:20:20.202154 13879 ts_tablet_manager.cc:1918] T 9255e8e7316a476a9ebafb666b0f5902 P 94cc42ad7f314dffab58a4f89620c868: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:20:20.202270 13879 log.cc:1196] T 9255e8e7316a476a9ebafb666b0f5902 P 94cc42ad7f314dffab58a4f89620c868: Deleting WAL directory at /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790626764250411-3144-0/minicluster-data/ts-1/wal/wals/9255e8e7316a476a9ebafb666b0f5902
I20260928 20:20:20.202605 13879 ts_tablet_manager.cc:1939] T 9255e8e7316a476a9ebafb666b0f5902 P 94cc42ad7f314dffab58a4f89620c868: Deleting consensus metadata
I20260928 20:20:20.203300 13363 catalog_manager.cc:4841] TS 94cc42ad7f314dffab58a4f89620c868 (127.3.18.2:43409): tablet 9255e8e7316a476a9ebafb666b0f5902 (table test-table [id=28458b5e9bb0402fac6b9d6452675318]) successfully deleted
I20260928 20:20:20.203918 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 13574
I20260928 20:20:20.212255 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 13708
I20260928 20:20:20.216825 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 13346
2026-09-28T20:20:20Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2214 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:20:20 dist-test-slave-b83v krb5kdc[13883](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:20:20 dist-test-slave-b83v krb5kdc[13883](info): set up 2 sockets
Sep 28 20:20:20 dist-test-slave-b83v krb5kdc[13883](info): commencing operation
krb5kdc: starting...
W20260928 20:20:21.297209 3144 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Sep 28 20:20:21 dist-test-slave-b83v krb5kdc[13883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:20:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:20:21Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.3.18.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.3.18.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.3.18.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/krb5kdc/rangeradmin_127.3.18.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.3.18.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/krb5kdc/rangeradmin_127.3.18.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.3.18.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.3.18.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.3.18.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/krb5kdc/rangerlookup_127.3.18.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.3.18.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/krb5kdc/rangerlookup_127.3.18.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.18.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.18.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.18.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/krb5kdc/HTTP_127.3.18.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.18.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/krb5kdc/HTTP_127.3.18.20@KRBTEST.COM.keytab.
I20260928 20:20:21.489008 3144 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task2yfqO7/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/postgres -l logfile start
2026-09-28 20:20:24.414 UTC [13913] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 20:20:24.414 UTC [13913] LOG: listening on IPv4 address "127.3.18.20", port 53343
2026-09-28 20:20:24.418 UTC [13913] LOG: listening on Unix socket "/tmp/.s.PGSQL.53343"
2026-09-28 20:20:24.440 UTC [13916] LOG: database system was shut down at 2026-09-28 20:20:22 UTC
2026-09-28 20:20:24.444 UTC [13913] LOG: database system is ready to accept connections
I20260928 20:20:25.409021 3144 mini_postgres.cc:96] Postgres bound to 53343
2026-09-28 20:20:25.417 UTC [13924] FATAL: database "slave" does not exist
127.3.18.20:53343 - accepting connections
I20260928 20:20:25.418210 3144 mini_ranger.cc:161] Starting Ranger...
I20260928 20:20:25.435321 3144 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 20:20:26.075109 3144 mini_ranger.cc:87] Created ranger Postgres database
I20260928 20:20:26.075206 3144 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/ranger-admin
2026-09-28 20:20:26,384 [I] DB FLAVOR :POSTGRES
2026-09-28 20:20:26,384 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 20:20:26,384 [I] Checking connection..
2026-09-28 20:20:26,385 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2yfqO7/build/release/bin/postgresql.jar:/tmp/dist-test-task2yfqO7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.18.20:53343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 20:20:26,660 [I] Checking connection passed.
2026-09-28 20:20:26,661 [I] --------- Verifying version history table ---------
2026-09-28 20:20:26,661 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2yfqO7/build/release/bin/postgresql.jar:/tmp/dist-test-task2yfqO7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.18.20:53343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 20:20:26,924 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 20:20:26,924 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2yfqO7/build/release/bin/postgresql.jar:/tmp/dist-test-task2yfqO7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.18.20:53343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 20:20:27,188 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 20:20:27,188 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 20:20:27,188 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2yfqO7/build/release/bin/postgresql.jar:/tmp/dist-test-task2yfqO7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.18.20:53343/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task2yfqO7/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 20:20:27.471 UTC [13988] WARNING: there is no transaction in progress
2026-09-28 20:20:27,477 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 20:20:27,478 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2yfqO7/build/release/bin/postgresql.jar:/tmp/dist-test-task2yfqO7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.18.20:53343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 20:20:27,748 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 20:20:27,749 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 20:20:27,749 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2yfqO7/build/release/bin/postgresql.jar:/tmp/dist-test-task2yfqO7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.18.20:53343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 20:20:27,987 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2yfqO7/build/release/bin/postgresql.jar:/tmp/dist-test-task2yfqO7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.18.20:53343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 20:20:28,233 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2yfqO7/build/release/bin/postgresql.jar:/tmp/dist-test-task2yfqO7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.18.20:53343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-b83v.c.gcp-upstream.internal','N') ;"
2026-09-28 20:20:28,478 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2yfqO7/build/release/bin/postgresql.jar:/tmp/dist-test-task2yfqO7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.18.20:53343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 20:20:28,721 [I] Table x_portal_user does not exist in database ranger
2026-09-28 20:20:28,721 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2yfqO7/build/release/bin/postgresql.jar:/tmp/dist-test-task2yfqO7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.18.20:53343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 20:20:28,972 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 20:20:28,973 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2yfqO7/build/release/bin/postgresql.jar:/tmp/dist-test-task2yfqO7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.18.20:53343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 20:20:29,236 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2yfqO7/build/release/bin/postgresql.jar:/tmp/dist-test-task2yfqO7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.18.20:53343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 20:20:29,506 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 20:20:29,506 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2yfqO7/build/release/bin/postgresql.jar:/tmp/dist-test-task2yfqO7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.18.20:53343/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task2yfqO7/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 20:20:30.638 UTC [14096] WARNING: there is no transaction in progress
2026-09-28 20:20:30.658 UTC [14096] WARNING: there is no transaction in progress
2026-09-28 20:20:30.676 UTC [14096] WARNING: there is no transaction in progress
2026-09-28 20:20:30.695 UTC [14096] WARNING: there is no transaction in progress
2026-09-28 20:20:30.715 UTC [14096] WARNING: there is no transaction in progress
2026-09-28 20:20:30.739 UTC [14096] WARNING: there is no transaction in progress
2026-09-28 20:20:30.809 UTC [14096] WARNING: there is no transaction in progress
2026-09-28 20:20:30.820 UTC [14096] WARNING: there is no transaction in progress
2026-09-28 20:20:30.837 UTC [14096] WARNING: there is no transaction in progress
2026-09-28 20:20:30.849 UTC [14096] WARNING: there is no transaction in progress
2026-09-28 20:20:30.862 UTC [14096] WARNING: there is no transaction in progress
2026-09-28 20:20:30.878 UTC [14096] WARNING: there is no transaction in progress
2026-09-28 20:20:30.890 UTC [14096] WARNING: there is no transaction in progress
2026-09-28 20:20:30.898 UTC [14096] WARNING: there is no transaction in progress
2026-09-28 20:20:30.906 UTC [14096] WARNING: there is no transaction in progress
2026-09-28 20:20:30.915 UTC [14096] WARNING: there is no transaction in progress
2026-09-28 20:20:31,741 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 20:20:31,742 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2yfqO7/build/release/bin/postgresql.jar:/tmp/dist-test-task2yfqO7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.18.20:53343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 20:20:31,987 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 20:20:31,987 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2yfqO7/build/release/bin/postgresql.jar:/tmp/dist-test-task2yfqO7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.18.20:53343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 20:20:32,247 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 20:20:32,247 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2yfqO7/build/release/bin/postgresql.jar:/tmp/dist-test-task2yfqO7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.18.20:53343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 20:20:32,501 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 20:20:32,501 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2yfqO7/build/release/bin/postgresql.jar:/tmp/dist-test-task2yfqO7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.18.20:53343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 20:20:32,735 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2yfqO7/build/release/bin/postgresql.jar:/tmp/dist-test-task2yfqO7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.18.20:53343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 20:20:32,999 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2yfqO7/build/release/bin/postgresql.jar:/tmp/dist-test-task2yfqO7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.18.20:53343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-b83v.c.gcp-upstream.internal';"
2026-09-28 20:20:33,242 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 20:20:33,242 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2yfqO7/build/release/bin/postgresql.jar:/tmp/dist-test-task2yfqO7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.18.20:53343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 20:20:33,489 [I] DB_PATCHES have already been applied
I20260928 20:20:33.495476 3144 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/ranger-admin:/tmp/dist-test-task2yfqO7/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task2yfqO7/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task2yfqO7/build/release/bin/postgresql.jar:/tmp/dist-test-task2yfqO7/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 20:20:33.495553 3144 mini_ranger.cc:193] Using host: 127.3.18.20
I20260928 20:20:33.499032 3144 mini_ranger.cc:227] Ranger admin URL: http://127.3.18.20:33807
Sep 28, 2026 8:20:33 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 8:20:33 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 8:20:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 8:20:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 8:20:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 8:20:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 8:20:34 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 20:20:34.638170 3144 mini_ranger.cc:160] Time spent starting Ranger: real 9.220s user 0.000s sys 0.010s
Sep 28, 2026 8:20:56 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 8:20:56 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 28, 2026 8:20:56 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 8:20:56 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 8:20:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 8:20:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 8:20:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 8:20:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 8:20:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 8:20:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 8:20:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 8:20:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 8:20:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 8:20:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 8:20:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 8:20:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 8:20:56 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 8:20:57 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 20:20:58.541517 3144 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.3.18.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.3.18.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.3.18.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/krb5kdc/rangerkms_127.3.18.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.3.18.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/krb5kdc/rangerkms_127.3.18.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.18.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.3.18.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.18.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/krb5kdc/HTTP_127.3.18.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.18.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/krb5kdc/HTTP_127.3.18.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 20:20:58 dist-test-slave-b83v krb5kdc[13883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626858, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 20:20:58.642999 3144 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 20:20:58.654650 3144 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 20:21:00.014540 3144 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 20:21:00.014632 3144 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/ranger-kms
I20260928 20:21:00.014665 3144 mini_ranger_kms.cc:227] Using postgres at 127.3.18.20:53343
2026-09-28 20:21:00,855 [I] DB FLAVOR :POSTGRES
2026-09-28 20:21:00,855 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 20:21:00,855 [I] Checking connection
2026-09-28 20:21:00,855 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2yfqO7/build/release/bin/postgresql.jar:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.18.20:53343/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 20:21:01,158 [I] connection success
2026-09-28 20:21:01,158 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 20:21:01,158 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 20:21:01,159 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2yfqO7/build/release/bin/postgresql.jar:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.18.20:53343/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 20:21:01,439 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 20:21:01,439 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 20:21:01,439 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 20:21:01,439 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2yfqO7/build/release/bin/postgresql.jar:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.18.20:53343/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 20:21:01,732 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 20:21:02.311894 3144 mini_ranger_kms.cc:310] Created kms service
I20260928 20:21:02.548511 3144 mini_ranger_kms.cc:326] Created kudu user
I20260928 20:21:02.600679 3144 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 20:21:02 dist-test-slave-b83v krb5kdc[13883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626858, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.3.18.20@KRBTEST.COM
I20260928 20:21:03.054098 3144 mini_ranger_kms.cc:384] Added ranger policy
I20260928 20:21:03.054214 3144 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/ranger-kms:/tmp/dist-test-task2yfqO7/build/release/bin/postgresql.jar:/tmp/dist-test-task2yfqO7/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task2yfqO7/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 20:21:03.054235 3144 mini_ranger_kms.cc:240] Using host: 127.3.18.20
I20260928 20:21:03.058151 3144 mini_ranger_kms.cc:276] Ranger KMS PID: 14277
I20260928 20:21:03.058286 3144 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.3.18.20:60649
Sep 28, 2026 8:21:04 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 8:21:04 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 8:21:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 8:21:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 8:21:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 8:21:04 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 28 20:21:05 dist-test-slave-b83v krb5kdc[13883](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790626865, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.18.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:21:05.401547 3144 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.759s user 0.002s sys 0.263s
I20260928 20:21:05.401665 3144 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 20:21:05.401748 3144 mini_ranger_kms.cc:401] 127.3.18.20:60649/kms/v1/keys
Sep 28 20:21:05 dist-test-slave-b83v krb5kdc[13883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626865, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.18.20@KRBTEST.COM for HTTP/127.3.18.20@KRBTEST.COM
Sep 28 20:21:06 dist-test-slave-b83v krb5kdc[13883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626865, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.18.20@KRBTEST.COM for HTTP/127.3.18.20@KRBTEST.COM
Sep 28 20:21:08 dist-test-slave-b83v krb5kdc[13883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.3.18.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab.
Entry for principal kudu/127.3.18.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab.
I20260928 20:21:08.583509 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:38987
--webserver_interface=127.3.18.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:44107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.18.62:38987
--ranger_config_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.18.20:60649/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:21:08.667562 14327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:21:08.667958 14327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:21:08.668002 14327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:21:08.669338 14327 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:21:08.669418 14327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:21:08.669440 14327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:21:08.669461 14327 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:21:08.669493 14327 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:21:08.671250 14327 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:44107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/master-0/wal
--ranger_kms_url=127.3.18.20:60649/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.18.62:38987
--ranger_config_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:38987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.18.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:21:08.671562 14327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:21:08.671878 14327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:21:08.677402 14335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:21:08.677424 14332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:21:08.677402 14333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:21:08.677594 14327 server_base.cc:1034] running on GCE node
I20260928 20:21:08.678005 14327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:21:08.678330 14327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:21:08.679507 14327 hybrid_clock.cc:648] HybridClock initialized: now 1790626868679483 us; error 43 us; skew 500 ppm
Sep 28 20:21:08 dist-test-slave-b83v krb5kdc[13883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626868, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:21:08.689322 14327 init.cc:377] Logged in from keytab as kudu/127.3.18.62@KRBTEST.COM (short username kudu)
I20260928 20:21:08.689862 14327 webserver.cc:460] Webserver started at http://127.3.18.62:39267/ using document root <none> and password file <none>
I20260928 20:21:08.690104 14327 fs_manager.cc:359] Metadata directory not provided
I20260928 20:21:08.690164 14327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:21:08.690327 14327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 20:21:08 dist-test-slave-b83v krb5kdc[13883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626868, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.62@KRBTEST.COM for HTTP/127.3.18.20@KRBTEST.COM
I20260928 20:21:08.745337 14327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/master-0/data/instance:
uuid: "1e0a55a0256147998ef65d1dc0c80250"
format_stamp: "Formatted at 2026-09-28 20:21:08 on dist-test-slave-b83v"
server_key: "41d8860b079e660ffbedb4c6f503e4a6"
server_key_iv: "166e57fbbb63e08be9e7ff95e63036a9"
server_key_version: "kuduclusterkey@0"
I20260928 20:21:08.745757 14327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/master-0/wal/instance:
uuid: "1e0a55a0256147998ef65d1dc0c80250"
format_stamp: "Formatted at 2026-09-28 20:21:08 on dist-test-slave-b83v"
server_key: "41d8860b079e660ffbedb4c6f503e4a6"
server_key_iv: "166e57fbbb63e08be9e7ff95e63036a9"
server_key_version: "kuduclusterkey@0"
I20260928 20:21:08.747237 14327 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:21:08.760994 14342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:21:08.761269 14327 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:21:08.761384 14327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/master-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/master-0/wal
uuid: "1e0a55a0256147998ef65d1dc0c80250"
format_stamp: "Formatted at 2026-09-28 20:21:08 on dist-test-slave-b83v"
server_key: "41d8860b079e660ffbedb4c6f503e4a6"
server_key_iv: "166e57fbbb63e08be9e7ff95e63036a9"
server_key_version: "kuduclusterkey@0"
I20260928 20:21:08.761509 14327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:21:08.773904 14327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:21:08.776094 14327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:21:08.776271 14327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:21:08.781780 14327 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.62:38987
I20260928 20:21:08.781824 14394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.62:38987 every 8 connection(s)
I20260928 20:21:08.782284 14327 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
I20260928 20:21:08.783017 14395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:21:08.785535 14395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e0a55a0256147998ef65d1dc0c80250: Bootstrap starting.
I20260928 20:21:08.786187 14395 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e0a55a0256147998ef65d1dc0c80250: Neither blocks nor log segments found. Creating new log.
I20260928 20:21:08.786500 14395 log.cc:824] T 00000000000000000000000000000000 P 1e0a55a0256147998ef65d1dc0c80250: Log is configured to *not* fsync() on all Append() calls
I20260928 20:21:08.787322 14395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e0a55a0256147998ef65d1dc0c80250: No bootstrap required, opened a new log
I20260928 20:21:08.789770 14395 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e0a55a0256147998ef65d1dc0c80250 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e0a55a0256147998ef65d1dc0c80250" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 38987 } }
I20260928 20:21:08.789928 14395 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e0a55a0256147998ef65d1dc0c80250 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:21:08.789954 14395 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e0a55a0256147998ef65d1dc0c80250 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e0a55a0256147998ef65d1dc0c80250, State: Initialized, Role: FOLLOWER
I20260928 20:21:08.790151 14395 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e0a55a0256147998ef65d1dc0c80250 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e0a55a0256147998ef65d1dc0c80250" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 38987 } }
I20260928 20:21:08.790228 14395 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e0a55a0256147998ef65d1dc0c80250 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:21:08.790297 14395 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e0a55a0256147998ef65d1dc0c80250 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:21:08.790359 14395 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e0a55a0256147998ef65d1dc0c80250 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:21:08.790488 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 14327
I20260928 20:21:08.790604 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/master-0/wal/instance
I20260928 20:21:08.790827 3144 external_mini_cluster.cc:1428] Setting key 6bf2ac212db44c25d1c79eecdf29ce8c
I20260928 20:21:08.791209 14395 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e0a55a0256147998ef65d1dc0c80250 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e0a55a0256147998ef65d1dc0c80250" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 38987 } }
I20260928 20:21:08.791385 14395 leader_election.cc:302] T 00000000000000000000000000000000 P 1e0a55a0256147998ef65d1dc0c80250 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e0a55a0256147998ef65d1dc0c80250; no voters:
I20260928 20:21:08.791623 14395 leader_election.cc:288] T 00000000000000000000000000000000 P 1e0a55a0256147998ef65d1dc0c80250 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:21:08.791815 14401 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e0a55a0256147998ef65d1dc0c80250 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:21:08.791949 14395 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e0a55a0256147998ef65d1dc0c80250 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:21:08.791978 14401 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e0a55a0256147998ef65d1dc0c80250 [term 1 LEADER]: Becoming Leader. State: Replica: 1e0a55a0256147998ef65d1dc0c80250, State: Running, Role: LEADER
I20260928 20:21:08.793010 14401 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e0a55a0256147998ef65d1dc0c80250 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e0a55a0256147998ef65d1dc0c80250" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 38987 } }
I20260928 20:21:08.793612 14400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e0a55a0256147998ef65d1dc0c80250 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e0a55a0256147998ef65d1dc0c80250. Latest consensus state: current_term: 1 leader_uuid: "1e0a55a0256147998ef65d1dc0c80250" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e0a55a0256147998ef65d1dc0c80250" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 38987 } } }
I20260928 20:21:08.793781 14400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e0a55a0256147998ef65d1dc0c80250 [sys.catalog]: This master's current role is: LEADER
I20260928 20:21:08.793761 14402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e0a55a0256147998ef65d1dc0c80250 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e0a55a0256147998ef65d1dc0c80250" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e0a55a0256147998ef65d1dc0c80250" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 38987 } } }
I20260928 20:21:08.793895 14402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e0a55a0256147998ef65d1dc0c80250 [sys.catalog]: This master's current role is: LEADER
I20260928 20:21:08.794106 14395 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 20:21:08.794416 14414 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:21:08.797250 14414 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 20:21:08 dist-test-slave-b83v krb5kdc[13883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:21:08.800674 14414 catalog_manager.cc:1347] Generated new cluster ID: 97912eba919145b3921834914673e706
I20260928 20:21:08.800730 14414 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:21:08.807912 14399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:21:08.791438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45976 (local address 127.3.18.62:38987)
0928 20:21:08.791642 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 20:21:08.791652 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 20:21:08.791696 (+ 44us) server_negotiation.cc:407] Connection header received
0928 20:21:08.791861 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:21:08.791869 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:21:08.791954 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:21:08.792249 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:21:08.792995 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:21:08.794430 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:21:08.795898 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:21:08.797613 (+ 1715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:21:08.797811 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:21:08.803899 (+ 6088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:21:08.803904 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:21:08.803905 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:21:08.803921 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:21:08.806088 (+ 2167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:21:08.806545 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:21:08.806548 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:21:08.806549 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:21:08.806650 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:21:08.806858 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:21:08.806860 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:21:08.806860 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:21:08.807172 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:21:08.807302 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 20:21:08.807425 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 20:21:08.807552 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":107,"threads_started":1}
I20260928 20:21:08.809865 14414 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:21:08.810364 14414 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:21:08.814415 14414 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e0a55a0256147998ef65d1dc0c80250: Generated new TSK 0
I20260928 20:21:08.814603 14414 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.18.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab.
Entry for principal kudu/127.3.18.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab.
I20260928 20:21:09.609169 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.18.20:60649/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.1:0
--local_ip_for_outbound_sockets=127.3.18.1
--webserver_interface=127.3.18.1
--webserver_port=0
--tserver_master_addrs=127.3.18.62:38987
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab
--principal=kudu/127.3.18.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:44107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:21:09.726052 14441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:21:09.726388 14441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:21:09.726444 14441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:21:09.728235 14441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:21:09.728302 14441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:21:09.728341 14441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.1
I20260928 20:21:09.730587 14441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:44107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.3.18.20:60649/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab
--principal=kudu/127.3.18.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.18.1
--webserver_port=0
--tserver_master_addrs=127.3.18.62:38987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14441
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.1
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:21:09.730885 14441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:21:09.731163 14441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:21:09.736692 14446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:21:09.736763 14447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:21:09.736864 14441 server_base.cc:1034] running on GCE node
W20260928 20:21:09.736696 14449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:21:09.737228 14441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:21:09.737493 14441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:21:09.738678 14441 hybrid_clock.cc:648] HybridClock initialized: now 1790626869738660 us; error 31 us; skew 500 ppm
Sep 28 20:21:09 dist-test-slave-b83v krb5kdc[13883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626869, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:21:09.742077 14441 init.cc:377] Logged in from keytab as kudu/127.3.18.1@KRBTEST.COM (short username kudu)
I20260928 20:21:09.742725 14441 webserver.cc:460] Webserver started at http://127.3.18.1:44675/ using document root <none> and password file <none>
I20260928 20:21:09.743033 14441 fs_manager.cc:359] Metadata directory not provided
I20260928 20:21:09.743106 14441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:21:09.743316 14441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 20:21:09 dist-test-slave-b83v krb5kdc[13883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626869, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.1@KRBTEST.COM for HTTP/127.3.18.20@KRBTEST.COM
I20260928 20:21:09.758646 14441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/ts-0/data/instance:
uuid: "d9faf48b19064ab8a541c3febc28898c"
format_stamp: "Formatted at 2026-09-28 20:21:09 on dist-test-slave-b83v"
server_key: "cf93948dac8e3faab4cd3b8848f87377"
server_key_iv: "4ec8fbc9df4fdddc51ec0db522c3493a"
server_key_version: "kuduclusterkey@0"
I20260928 20:21:09.759093 14441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/ts-0/wal/instance:
uuid: "d9faf48b19064ab8a541c3febc28898c"
format_stamp: "Formatted at 2026-09-28 20:21:09 on dist-test-slave-b83v"
server_key: "cf93948dac8e3faab4cd3b8848f87377"
server_key_iv: "4ec8fbc9df4fdddc51ec0db522c3493a"
server_key_version: "kuduclusterkey@0"
I20260928 20:21:09.771991 14441 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.004s sys 0.000s
I20260928 20:21:09.786747 14458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:21:09.786988 14441 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 20:21:09.787098 14441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/ts-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/ts-0/wal
uuid: "d9faf48b19064ab8a541c3febc28898c"
format_stamp: "Formatted at 2026-09-28 20:21:09 on dist-test-slave-b83v"
server_key: "cf93948dac8e3faab4cd3b8848f87377"
server_key_iv: "4ec8fbc9df4fdddc51ec0db522c3493a"
server_key_version: "kuduclusterkey@0"
I20260928 20:21:09.787197 14441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:21:09.807839 14441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:21:09.810473 14441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:21:09.810695 14441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:21:09.810994 14441 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:21:09.811401 14441 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:21:09.811486 14441 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:21:09.811578 14441 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:21:09.811659 14441 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:21:09.818394 14441 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.1:38105
I20260928 20:21:09.818579 14571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.1:38105 every 8 connection(s)
I20260928 20:21:09.818868 14441 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
Sep 28 20:21:09 dist-test-slave-b83v krb5kdc[13883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626869, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.1@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:21:09.826606 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 14441
I20260928 20:21:09.826740 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/ts-0/wal/instance
I20260928 20:21:09.826960 3144 external_mini_cluster.cc:1428] Setting key e5b9bea786a415809ee711a262d2595d
I20260928 20:21:09.831368 14574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:21:09.819739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:38987 (local address 127.3.18.1:45963)
0928 20:21:09.819927 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 20:21:09.819981 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 20:21:09.820682 (+ 701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:21:09.821000 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:21:09.821005 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:21:09.821290 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:21:09.821581 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:21:09.821589 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:21:09.822350 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:21:09.822352 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:21:09.823649 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:21:09.823652 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:21:09.824153 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:21:09.824154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:21:09.824304 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:21:09.824927 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:21:09.824936 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:21:09.826617 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:21:09.829341 (+ 2724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:21:09.829351 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:21:09.829353 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:21:09.829735 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:21:09.830042 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:21:09.830043 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:21:09.830044 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:21:09.830230 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:21:09.830579 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:21:09.830587 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:21:09.830799 (+ 212us) client_negotiation.cc:769] Sending connection context
0928 20:21:09.830915 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 20:21:09.831155 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":84,"threads_started":1}
I20260928 20:21:09.831847 14575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:21:09.820062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:45963 (local address 127.3.18.62:38987)
0928 20:21:09.820166 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0928 20:21:09.820169 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:21:09.820179 (+ 10us) server_negotiation.cc:407] Connection header received
0928 20:21:09.820819 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:21:09.820820 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:21:09.820836 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:21:09.820909 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:21:09.821694 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:21:09.822244 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:21:09.823755 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:21:09.824037 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:21:09.824136 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:21:09.826763 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:21:09.826765 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:21:09.826766 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:21:09.826775 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:21:09.829229 (+ 2454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:21:09.829845 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:21:09.829846 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:21:09.829847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:21:09.829928 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:21:09.830325 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:21:09.830326 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:21:09.830326 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:21:09.830493 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:21:09.831550 (+ 1057us) server_negotiation.cc:1035] Waiting for connection context
0928 20:21:09.831605 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 20:21:09.831709 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":53,"threads_started":1}
I20260928 20:21:09.832396 14572 heartbeater.cc:347] Connected to a master server at 127.3.18.62:38987
I20260928 20:21:09.832548 14572 heartbeater.cc:464] Registering TS with master...
I20260928 20:21:09.832782 14572 heartbeater.cc:511] Master 127.3.18.62:38987 requested a full tablet report, sending...
I20260928 20:21:09.833380 14359 ts_manager.cc:194] Registered new tserver with Master: d9faf48b19064ab8a541c3febc28898c (127.3.18.1:38105)
I20260928 20:21:09.834102 14359 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.18.1@KRBTEST.COM'} at 127.3.18.1:45963
WARNING: no policy specified for kudu/127.3.18.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab.
Entry for principal kudu/127.3.18.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab.
I20260928 20:21:09.863716 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.18.20:60649/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.2:0
--local_ip_for_outbound_sockets=127.3.18.2
--webserver_interface=127.3.18.2
--webserver_port=0
--tserver_master_addrs=127.3.18.62:38987
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab
--principal=kudu/127.3.18.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:44107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:21:09.963099 14578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:21:09.963377 14578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:21:09.963421 14578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:21:09.965044 14578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:21:09.965122 14578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:21:09.965170 14578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.2
Sep 28 20:21:09 dist-test-slave-b83v krb5kdc[13883](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790626869, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:21:09.967162 14578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:44107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.3.18.20:60649/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab
--principal=kudu/127.3.18.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.18.2
--webserver_port=0
--tserver_master_addrs=127.3.18.62:38987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.2
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:21:09.967443 14578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:21:09.967725 14578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:21:09.974239 14586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:21:09.974263 14588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:21:09.974517 14585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:21:09.976214 14578 server_base.cc:1034] running on GCE node
I20260928 20:21:09.976526 14578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:21:09.976871 14578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:21:09.978067 14578 hybrid_clock.cc:648] HybridClock initialized: now 1790626869978039 us; error 42 us; skew 500 ppm
Sep 28 20:21:09 dist-test-slave-b83v krb5kdc[13883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626869, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:21:09.981448 14578 init.cc:377] Logged in from keytab as kudu/127.3.18.2@KRBTEST.COM (short username kudu)
I20260928 20:21:09.982089 14578 webserver.cc:460] Webserver started at http://127.3.18.2:44399/ using document root <none> and password file <none>
I20260928 20:21:09.982365 14578 fs_manager.cc:359] Metadata directory not provided
I20260928 20:21:09.982422 14578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:21:09.982556 14578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 20:21:09 dist-test-slave-b83v krb5kdc[13883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626869, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.2@KRBTEST.COM for HTTP/127.3.18.20@KRBTEST.COM
I20260928 20:21:10.002271 14578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/ts-1/data/instance:
uuid: "e88c0e7b03e14d61b08ecda72422cd0e"
format_stamp: "Formatted at 2026-09-28 20:21:09 on dist-test-slave-b83v"
server_key: "2993f11cd92da6560e52f1ce496003fd"
server_key_iv: "fc8aab2cc10e1d1a137febe40c6b0e36"
server_key_version: "kuduclusterkey@0"
I20260928 20:21:10.002825 14578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/ts-1/wal/instance:
uuid: "e88c0e7b03e14d61b08ecda72422cd0e"
format_stamp: "Formatted at 2026-09-28 20:21:09 on dist-test-slave-b83v"
server_key: "2993f11cd92da6560e52f1ce496003fd"
server_key_iv: "fc8aab2cc10e1d1a137febe40c6b0e36"
server_key_version: "kuduclusterkey@0"
I20260928 20:21:10.012130 14578 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.000s sys 0.004s
I20260928 20:21:10.027944 14595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:21:10.028214 14578 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 20:21:10.028316 14578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/ts-1/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/ts-1/wal
uuid: "e88c0e7b03e14d61b08ecda72422cd0e"
format_stamp: "Formatted at 2026-09-28 20:21:09 on dist-test-slave-b83v"
server_key: "2993f11cd92da6560e52f1ce496003fd"
server_key_iv: "fc8aab2cc10e1d1a137febe40c6b0e36"
server_key_version: "kuduclusterkey@0"
I20260928 20:21:10.028419 14578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:21:10.051484 14578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:21:10.054443 14578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:21:10.054656 14578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:21:10.054950 14578 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:21:10.055389 14578 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:21:10.055451 14578 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:21:10.055537 14578 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:21:10.055572 14578 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:21:10.064931 14578 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.2:34261
I20260928 20:21:10.065270 14709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.2:34261 every 8 connection(s)
I20260928 20:21:10.065668 14578 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
I20260928 20:21:10.072175 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 14578
I20260928 20:21:10.072307 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/ts-1/wal/instance
I20260928 20:21:10.072502 3144 external_mini_cluster.cc:1428] Setting key 03b9db36f3078c7c2478dbe4634a29d7
Sep 28 20:21:10 dist-test-slave-b83v krb5kdc[13883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626869, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.2@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:21:10.080013 14575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:21:10.066718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:54903 (local address 127.3.18.62:38987)
0928 20:21:10.066810 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0928 20:21:10.066813 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:21:10.067065 (+ 252us) server_negotiation.cc:407] Connection header received
0928 20:21:10.067854 (+ 789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:21:10.067856 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:21:10.067870 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:21:10.067945 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:21:10.068791 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:21:10.069626 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:21:10.071360 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:21:10.071702 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:21:10.071820 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:21:10.074873 (+ 3053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:21:10.074874 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:21:10.074875 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:21:10.074884 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:21:10.078039 (+ 3155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:21:10.078692 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:21:10.078694 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:21:10.078694 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:21:10.078777 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:21:10.079095 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:21:10.079096 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:21:10.079096 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:21:10.079243 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:21:10.079289 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 20:21:10.079750 (+ 461us) server_negotiation.cc:299] Negotiation successful
0928 20:21:10.079900 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 20:21:10.080888 14712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:21:10.066700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:38987 (local address 127.3.18.2:54903)
0928 20:21:10.066892 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 20:21:10.066949 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 20:21:10.067667 (+ 718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:21:10.068035 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:21:10.068040 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:21:10.068341 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:21:10.068659 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:21:10.068666 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:21:10.069730 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:21:10.069739 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:21:10.071219 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:21:10.071223 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:21:10.071852 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:21:10.071854 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:21:10.072027 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:21:10.072726 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:21:10.072750 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:21:10.074611 (+ 1861us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:21:10.078177 (+ 3566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:21:10.078190 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:21:10.078193 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:21:10.078562 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:21:10.078852 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:21:10.078852 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:21:10.078853 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:21:10.079019 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:21:10.079339 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:21:10.079346 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:21:10.079558 (+ 212us) client_negotiation.cc:769] Sending connection context
0928 20:21:10.080432 (+ 874us) client_negotiation.cc:240] Negotiation successful
0928 20:21:10.080625 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":84,"threads_started":1}
I20260928 20:21:10.081843 14710 heartbeater.cc:347] Connected to a master server at 127.3.18.62:38987
I20260928 20:21:10.081960 14710 heartbeater.cc:464] Registering TS with master...
I20260928 20:21:10.082175 14710 heartbeater.cc:511] Master 127.3.18.62:38987 requested a full tablet report, sending...
I20260928 20:21:10.082612 14359 ts_manager.cc:194] Registered new tserver with Master: e88c0e7b03e14d61b08ecda72422cd0e (127.3.18.2:34261)
I20260928 20:21:10.083163 14359 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.18.2@KRBTEST.COM'} at 127.3.18.2:54903
WARNING: no policy specified for kudu/127.3.18.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab.
Entry for principal kudu/127.3.18.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab.
I20260928 20:21:10.108201 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.18.20:60649/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.3:0
--local_ip_for_outbound_sockets=127.3.18.3
--webserver_interface=127.3.18.3
--webserver_port=0
--tserver_master_addrs=127.3.18.62:38987
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab
--principal=kudu/127.3.18.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:44107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:21:10.231330 14715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:21:10.231644 14715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:21:10.231681 14715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:21:10.233320 14715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:21:10.233383 14715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:21:10.233415 14715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.3
I20260928 20:21:10.235167 14715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:44107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.3.18.20:60649/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab
--principal=kudu/127.3.18.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.18.3
--webserver_port=0
--tserver_master_addrs=127.3.18.62:38987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.3
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:21:10.235477 14715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:21:10.235755 14715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:21:10.241305 14720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:21:10.241446 14723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:21:10.241328 14721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:21:10.241791 14715 server_base.cc:1034] running on GCE node
I20260928 20:21:10.242018 14715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:21:10.242323 14715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:21:10.243517 14715 hybrid_clock.cc:648] HybridClock initialized: now 1790626870243496 us; error 33 us; skew 500 ppm
Sep 28 20:21:10 dist-test-slave-b83v krb5kdc[13883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626870, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:21:10.246937 14715 init.cc:377] Logged in from keytab as kudu/127.3.18.3@KRBTEST.COM (short username kudu)
I20260928 20:21:10.247417 14715 webserver.cc:460] Webserver started at http://127.3.18.3:46211/ using document root <none> and password file <none>
I20260928 20:21:10.247684 14715 fs_manager.cc:359] Metadata directory not provided
I20260928 20:21:10.247759 14715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:21:10.247970 14715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 20:21:10 dist-test-slave-b83v krb5kdc[13883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626870, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.3@KRBTEST.COM for HTTP/127.3.18.20@KRBTEST.COM
I20260928 20:21:10.273902 14715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/ts-2/data/instance:
uuid: "95f0fbb445d148c597f20f823517cab2"
format_stamp: "Formatted at 2026-09-28 20:21:10 on dist-test-slave-b83v"
server_key: "515880d23ac9394022047a321287077f"
server_key_iv: "53e0533781e13385f288b78d849c5d96"
server_key_version: "kuduclusterkey@0"
I20260928 20:21:10.274494 14715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/ts-2/wal/instance:
uuid: "95f0fbb445d148c597f20f823517cab2"
format_stamp: "Formatted at 2026-09-28 20:21:10 on dist-test-slave-b83v"
server_key: "515880d23ac9394022047a321287077f"
server_key_iv: "53e0533781e13385f288b78d849c5d96"
server_key_version: "kuduclusterkey@0"
I20260928 20:21:10.276881 14715 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 20:21:10.302021 14730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:21:10.302317 14715 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260928 20:21:10.302418 14715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/ts-2/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/ts-2/wal
uuid: "95f0fbb445d148c597f20f823517cab2"
format_stamp: "Formatted at 2026-09-28 20:21:10 on dist-test-slave-b83v"
server_key: "515880d23ac9394022047a321287077f"
server_key_iv: "53e0533781e13385f288b78d849c5d96"
server_key_version: "kuduclusterkey@0"
I20260928 20:21:10.302520 14715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:21:10.320996 14715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:21:10.323604 14715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:21:10.323866 14715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:21:10.324205 14715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:21:10.324649 14715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:21:10.324759 14715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:21:10.324841 14715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:21:10.324904 14715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:21:10.331382 14715 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.3:39491
I20260928 20:21:10.331970 14715 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
I20260928 20:21:10.334787 14843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.3:39491 every 8 connection(s)
I20260928 20:21:10.338938 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 14715
I20260928 20:21:10.339061 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/ts-2/wal/instance
I20260928 20:21:10.339258 3144 external_mini_cluster.cc:1428] Setting key 7b72aaf810e3136a082e501838ad2d55
Sep 28 20:21:10 dist-test-slave-b83v krb5kdc[13883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626870, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.3@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:21:10.348074 14575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:21:10.335730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:34231 (local address 127.3.18.62:38987)
0928 20:21:10.335795 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 20:21:10.335797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:21:10.336322 (+ 525us) server_negotiation.cc:407] Connection header received
0928 20:21:10.337106 (+ 784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:21:10.337108 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:21:10.337122 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:21:10.337157 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:21:10.338119 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:21:10.338902 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:21:10.341192 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:21:10.341493 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:21:10.341540 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:21:10.344508 (+ 2968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:21:10.344510 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:21:10.344510 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:21:10.344518 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:21:10.346351 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:21:10.346890 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:21:10.346891 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:21:10.346892 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:21:10.346952 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:21:10.347215 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:21:10.347216 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:21:10.347217 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:21:10.347354 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:21:10.347403 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 20:21:10.347883 (+ 480us) server_negotiation.cc:299] Negotiation successful
0928 20:21:10.347972 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 20:21:10.348932 14846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:21:10.335404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:38987 (local address 127.3.18.3:34231)
0928 20:21:10.336068 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0928 20:21:10.336230 (+ 162us) client_negotiation.cc:174] Beginning negotiation
0928 20:21:10.336911 (+ 681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:21:10.337367 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:21:10.337371 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:21:10.337644 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:21:10.337957 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:21:10.337964 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:21:10.339034 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:21:10.339035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:21:10.341086 (+ 2051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:21:10.341089 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:21:10.341634 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:21:10.341634 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:21:10.341797 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:21:10.342574 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:21:10.342584 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:21:10.344387 (+ 1803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:21:10.346457 (+ 2070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:21:10.346465 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:21:10.346469 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:21:10.346798 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:21:10.347016 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:21:10.347016 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:21:10.347017 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:21:10.347149 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:21:10.347434 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:21:10.347439 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:21:10.347624 (+ 185us) client_negotiation.cc:769] Sending connection context
0928 20:21:10.348469 (+ 845us) client_negotiation.cc:240] Negotiation successful
0928 20:21:10.348659 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":106,"threads_started":1}
I20260928 20:21:10.349629 14844 heartbeater.cc:347] Connected to a master server at 127.3.18.62:38987
I20260928 20:21:10.349745 14844 heartbeater.cc:464] Registering TS with master...
I20260928 20:21:10.349946 14844 heartbeater.cc:511] Master 127.3.18.62:38987 requested a full tablet report, sending...
I20260928 20:21:10.350402 14359 ts_manager.cc:194] Registered new tserver with Master: 95f0fbb445d148c597f20f823517cab2 (127.3.18.3:39491)
I20260928 20:21:10.350922 14359 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.18.3@KRBTEST.COM'} at 127.3.18.3:34231
I20260928 20:21:10.352559 3144 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 20:21:10.370671 14575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:21:10.360891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45984 (local address 127.3.18.62:38987)
0928 20:21:10.360950 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 20:21:10.360953 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:21:10.361252 (+ 299us) server_negotiation.cc:407] Connection header received
0928 20:21:10.361360 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:21:10.361361 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:21:10.361374 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:21:10.361454 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:21:10.363920 (+ 2466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:21:10.364480 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:21:10.365682 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:21:10.365965 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:21:10.366020 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:21:10.367615 (+ 1595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:21:10.367617 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:21:10.367617 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:21:10.367627 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:21:10.369346 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:21:10.369805 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:21:10.369807 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:21:10.369808 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:21:10.369892 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:21:10.370109 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:21:10.370110 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:21:10.370111 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:21:10.370258 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:21:10.370369 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 20:21:10.370475 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 20:21:10.370556 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 20:21:10.372674 14359 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45984:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 20:21:10.375185 14359 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 20:21:10.385006 14856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:21:10.378750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.2:34261 (local address 127.0.0.1:37762)
0928 20:21:10.380335 (+ 1585us) negotiation.cc:107] Waiting for socket to connect
0928 20:21:10.380346 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:21:10.380414 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:21:10.380869 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:21:10.380870 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:21:10.380873 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:21:10.380916 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:21:10.380918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:21:10.382230 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:21:10.382232 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:21:10.383560 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:21:10.383564 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:21:10.384706 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:21:10.384707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:21:10.384782 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:21:10.384799 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 20:21:10.384865 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 20:21:10.384915 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1569,"thread_start_us":105,"threads_started":1}
I20260928 20:21:10.385941 14860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:21:10.379627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37762 (local address 127.3.18.2:34261)
0928 20:21:10.379846 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 20:21:10.379850 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 20:21:10.380411 (+ 561us) server_negotiation.cc:407] Connection header received
0928 20:21:10.380749 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:21:10.380752 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:21:10.380765 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:21:10.380854 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:21:10.381199 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:21:10.382012 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:21:10.383835 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:21:10.384635 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:21:10.384708 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:21:10.384798 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 20:21:10.385788 (+ 990us) server_negotiation.cc:299] Negotiation successful
0928 20:21:10.385837 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":96,"threads_started":1}
I20260928 20:21:10.386838 14644 tablet_service.cc:1467] Processing CreateTablet for tablet 95f1dc70a42b43da9a431e2c63c15ece (DEFAULT_TABLE table=test-table [id=9b4aebef86c64e91a8cc1882674a35b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:21:10.387279 14644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95f1dc70a42b43da9a431e2c63c15ece. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:21:10.389801 14858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:21:10.378744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.1:38105 (local address 127.0.0.1:52526)
0928 20:21:10.379165 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0928 20:21:10.379214 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 20:21:10.379305 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:21:10.380136 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:21:10.380140 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:21:10.380143 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:21:10.380225 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:21:10.380228 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:21:10.387252 (+ 7024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:21:10.387253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:21:10.388404 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:21:10.388407 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:21:10.389525 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:21:10.389526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:21:10.389595 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:21:10.389600 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 20:21:10.389631 (+ 31us) client_negotiation.cc:240] Negotiation successful
0928 20:21:10.389676 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":109,"threads_started":1}
I20260928 20:21:10.390162 14861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:21:10.379850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52526 (local address 127.3.18.1:38105)
0928 20:21:10.379951 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0928 20:21:10.379955 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 20:21:10.379967 (+ 12us) server_negotiation.cc:407] Connection header received
0928 20:21:10.380007 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:21:10.380009 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:21:10.380027 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:21:10.380313 (+ 286us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:21:10.380433 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:21:10.387143 (+ 6710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:21:10.388526 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:21:10.389414 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:21:10.389932 (+ 518us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:21:10.390015 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 20:21:10.390045 (+ 30us) server_negotiation.cc:299] Negotiation successful
0928 20:21:10.390093 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78,"thread_start_us":58,"threads_started":1}
I20260928 20:21:10.391058 14506 tablet_service.cc:1467] Processing CreateTablet for tablet 95f1dc70a42b43da9a431e2c63c15ece (DEFAULT_TABLE table=test-table [id=9b4aebef86c64e91a8cc1882674a35b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:21:10.391628 14506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95f1dc70a42b43da9a431e2c63c15ece. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:21:10.394327 14859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:21:10.379392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53750 (local address 127.3.18.3:39491)
0928 20:21:10.379577 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 20:21:10.379583 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 20:21:10.380934 (+ 1351us) server_negotiation.cc:407] Connection header received
0928 20:21:10.380979 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:21:10.380982 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:21:10.381002 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:21:10.381096 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:21:10.382110 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:21:10.383220 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:21:10.385977 (+ 2757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:21:10.392347 (+ 6370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:21:10.392407 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:21:10.392478 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 20:21:10.394184 (+ 1706us) server_negotiation.cc:299] Negotiation successful
0928 20:21:10.394246 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":127,"threads_started":1}
I20260928 20:21:10.394632 14857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:21:10.379008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.3:39491 (local address 127.0.0.1:53750)
0928 20:21:10.380495 (+ 1487us) negotiation.cc:107] Waiting for socket to connect
0928 20:21:10.380503 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 20:21:10.380541 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:21:10.381094 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:21:10.381095 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:21:10.381097 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:21:10.381141 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:21:10.381142 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:21:10.383644 (+ 2502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:21:10.383645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:21:10.385856 (+ 2211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:21:10.385860 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:21:10.393611 (+ 7751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:21:10.393612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:21:10.394065 (+ 453us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:21:10.394071 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 20:21:10.394463 (+ 392us) client_negotiation.cc:240] Negotiation successful
0928 20:21:10.394523 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1474,"thread_start_us":78,"threads_started":1}
I20260928 20:21:10.395390 14778 tablet_service.cc:1467] Processing CreateTablet for tablet 95f1dc70a42b43da9a431e2c63c15ece (DEFAULT_TABLE table=test-table [id=9b4aebef86c64e91a8cc1882674a35b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:21:10.395634 14863 tablet_bootstrap.cc:492] T 95f1dc70a42b43da9a431e2c63c15ece P d9faf48b19064ab8a541c3febc28898c: Bootstrap starting.
I20260928 20:21:10.395720 14778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95f1dc70a42b43da9a431e2c63c15ece. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:21:10.396389 14863 tablet_bootstrap.cc:654] T 95f1dc70a42b43da9a431e2c63c15ece P d9faf48b19064ab8a541c3febc28898c: Neither blocks nor log segments found. Creating new log.
I20260928 20:21:10.396679 14863 log.cc:824] T 95f1dc70a42b43da9a431e2c63c15ece P d9faf48b19064ab8a541c3febc28898c: Log is configured to *not* fsync() on all Append() calls
I20260928 20:21:10.399170 14865 tablet_bootstrap.cc:492] T 95f1dc70a42b43da9a431e2c63c15ece P 95f0fbb445d148c597f20f823517cab2: Bootstrap starting.
I20260928 20:21:10.399514 14862 tablet_bootstrap.cc:492] T 95f1dc70a42b43da9a431e2c63c15ece P e88c0e7b03e14d61b08ecda72422cd0e: Bootstrap starting.
I20260928 20:21:10.400707 14862 tablet_bootstrap.cc:654] T 95f1dc70a42b43da9a431e2c63c15ece P e88c0e7b03e14d61b08ecda72422cd0e: Neither blocks nor log segments found. Creating new log.
I20260928 20:21:10.400707 14865 tablet_bootstrap.cc:654] T 95f1dc70a42b43da9a431e2c63c15ece P 95f0fbb445d148c597f20f823517cab2: Neither blocks nor log segments found. Creating new log.
I20260928 20:21:10.401100 14865 log.cc:824] T 95f1dc70a42b43da9a431e2c63c15ece P 95f0fbb445d148c597f20f823517cab2: Log is configured to *not* fsync() on all Append() calls
I20260928 20:21:10.401146 14862 log.cc:824] T 95f1dc70a42b43da9a431e2c63c15ece P e88c0e7b03e14d61b08ecda72422cd0e: Log is configured to *not* fsync() on all Append() calls
I20260928 20:21:10.402124 14863 tablet_bootstrap.cc:492] T 95f1dc70a42b43da9a431e2c63c15ece P d9faf48b19064ab8a541c3febc28898c: No bootstrap required, opened a new log
I20260928 20:21:10.402263 14863 ts_tablet_manager.cc:1397] T 95f1dc70a42b43da9a431e2c63c15ece P d9faf48b19064ab8a541c3febc28898c: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260928 20:21:10.405599 14863 raft_consensus.cc:348] T 95f1dc70a42b43da9a431e2c63c15ece P d9faf48b19064ab8a541c3febc28898c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e88c0e7b03e14d61b08ecda72422cd0e" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34261 } } peers { permanent_uuid: "d9faf48b19064ab8a541c3febc28898c" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 38105 } } peers { permanent_uuid: "95f0fbb445d148c597f20f823517cab2" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 39491 } }
I20260928 20:21:10.406021 14863 raft_consensus.cc:374] T 95f1dc70a42b43da9a431e2c63c15ece P d9faf48b19064ab8a541c3febc28898c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:21:10.406131 14863 raft_consensus.cc:729] T 95f1dc70a42b43da9a431e2c63c15ece P d9faf48b19064ab8a541c3febc28898c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9faf48b19064ab8a541c3febc28898c, State: Initialized, Role: FOLLOWER
I20260928 20:21:10.406406 14863 consensus_queue.cc:260] T 95f1dc70a42b43da9a431e2c63c15ece P d9faf48b19064ab8a541c3febc28898c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e88c0e7b03e14d61b08ecda72422cd0e" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34261 } } peers { permanent_uuid: "d9faf48b19064ab8a541c3febc28898c" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 38105 } } peers { permanent_uuid: "95f0fbb445d148c597f20f823517cab2" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 39491 } }
I20260928 20:21:10.406869 14863 ts_tablet_manager.cc:1428] T 95f1dc70a42b43da9a431e2c63c15ece P d9faf48b19064ab8a541c3febc28898c: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260928 20:21:10.407135 14572 heartbeater.cc:503] Master 127.3.18.62:38987 was elected leader, sending a full tablet report...
I20260928 20:21:10.408820 14865 tablet_bootstrap.cc:492] T 95f1dc70a42b43da9a431e2c63c15ece P 95f0fbb445d148c597f20f823517cab2: No bootstrap required, opened a new log
I20260928 20:21:10.408836 14862 tablet_bootstrap.cc:492] T 95f1dc70a42b43da9a431e2c63c15ece P e88c0e7b03e14d61b08ecda72422cd0e: No bootstrap required, opened a new log
I20260928 20:21:10.408910 14865 ts_tablet_manager.cc:1397] T 95f1dc70a42b43da9a431e2c63c15ece P 95f0fbb445d148c597f20f823517cab2: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260928 20:21:10.409101 14862 ts_tablet_manager.cc:1397] T 95f1dc70a42b43da9a431e2c63c15ece P e88c0e7b03e14d61b08ecda72422cd0e: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260928 20:21:10.411041 14865 raft_consensus.cc:348] T 95f1dc70a42b43da9a431e2c63c15ece P 95f0fbb445d148c597f20f823517cab2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e88c0e7b03e14d61b08ecda72422cd0e" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34261 } } peers { permanent_uuid: "d9faf48b19064ab8a541c3febc28898c" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 38105 } } peers { permanent_uuid: "95f0fbb445d148c597f20f823517cab2" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 39491 } }
I20260928 20:21:10.411222 14865 raft_consensus.cc:374] T 95f1dc70a42b43da9a431e2c63c15ece P 95f0fbb445d148c597f20f823517cab2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:21:10.411268 14865 raft_consensus.cc:729] T 95f1dc70a42b43da9a431e2c63c15ece P 95f0fbb445d148c597f20f823517cab2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95f0fbb445d148c597f20f823517cab2, State: Initialized, Role: FOLLOWER
I20260928 20:21:10.411401 14865 consensus_queue.cc:260] T 95f1dc70a42b43da9a431e2c63c15ece P 95f0fbb445d148c597f20f823517cab2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e88c0e7b03e14d61b08ecda72422cd0e" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34261 } } peers { permanent_uuid: "d9faf48b19064ab8a541c3febc28898c" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 38105 } } peers { permanent_uuid: "95f0fbb445d148c597f20f823517cab2" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 39491 } }
I20260928 20:21:10.411798 14862 raft_consensus.cc:348] T 95f1dc70a42b43da9a431e2c63c15ece P e88c0e7b03e14d61b08ecda72422cd0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e88c0e7b03e14d61b08ecda72422cd0e" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34261 } } peers { permanent_uuid: "d9faf48b19064ab8a541c3febc28898c" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 38105 } } peers { permanent_uuid: "95f0fbb445d148c597f20f823517cab2" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 39491 } }
I20260928 20:21:10.412027 14862 raft_consensus.cc:374] T 95f1dc70a42b43da9a431e2c63c15ece P e88c0e7b03e14d61b08ecda72422cd0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:21:10.412107 14865 ts_tablet_manager.cc:1428] T 95f1dc70a42b43da9a431e2c63c15ece P 95f0fbb445d148c597f20f823517cab2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 20:21:10.412109 14862 raft_consensus.cc:729] T 95f1dc70a42b43da9a431e2c63c15ece P e88c0e7b03e14d61b08ecda72422cd0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e88c0e7b03e14d61b08ecda72422cd0e, State: Initialized, Role: FOLLOWER
I20260928 20:21:10.412227 14844 heartbeater.cc:503] Master 127.3.18.62:38987 was elected leader, sending a full tablet report...
I20260928 20:21:10.412375 14862 consensus_queue.cc:260] T 95f1dc70a42b43da9a431e2c63c15ece P e88c0e7b03e14d61b08ecda72422cd0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e88c0e7b03e14d61b08ecda72422cd0e" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34261 } } peers { permanent_uuid: "d9faf48b19064ab8a541c3febc28898c" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 38105 } } peers { permanent_uuid: "95f0fbb445d148c597f20f823517cab2" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 39491 } }
I20260928 20:21:10.412710 14862 ts_tablet_manager.cc:1428] T 95f1dc70a42b43da9a431e2c63c15ece P e88c0e7b03e14d61b08ecda72422cd0e: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260928 20:21:10.412868 14710 heartbeater.cc:503] Master 127.3.18.62:38987 was elected leader, sending a full tablet report...
Sep 28 20:21:10 dist-test-slave-b83v krb5kdc[13883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626869, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.62@KRBTEST.COM for HTTP/127.3.18.20@KRBTEST.COM
I20260928 20:21:10.484120 14867 raft_consensus.cc:482] T 95f1dc70a42b43da9a431e2c63c15ece P d9faf48b19064ab8a541c3febc28898c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:21:10.484297 14867 raft_consensus.cc:504] T 95f1dc70a42b43da9a431e2c63c15ece P d9faf48b19064ab8a541c3febc28898c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e88c0e7b03e14d61b08ecda72422cd0e" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34261 } } peers { permanent_uuid: "d9faf48b19064ab8a541c3febc28898c" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 38105 } } peers { permanent_uuid: "95f0fbb445d148c597f20f823517cab2" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 39491 } }
I20260928 20:21:10.484707 14867 leader_election.cc:288] T 95f1dc70a42b43da9a431e2c63c15ece P d9faf48b19064ab8a541c3febc28898c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e88c0e7b03e14d61b08ecda72422cd0e (127.3.18.2:34261), 95f0fbb445d148c597f20f823517cab2 (127.3.18.3:39491)
I20260928 20:21:10.493204 14859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:21:10.485178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:59539 (local address 127.3.18.3:39491)
0928 20:21:10.485240 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 20:21:10.485242 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:21:10.488032 (+ 2790us) server_negotiation.cc:407] Connection header received
0928 20:21:10.488154 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:21:10.488156 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:21:10.488170 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:21:10.488212 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:21:10.488470 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:21:10.489311 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:21:10.492112 (+ 2801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:21:10.492759 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:21:10.492835 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:21:10.492847 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 20:21:10.493052 (+ 205us) server_negotiation.cc:299] Negotiation successful
0928 20:21:10.493105 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 20:21:10.493546 14871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:21:10.484960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.3:39491 (local address 127.3.18.1:59539)
0928 20:21:10.487942 (+ 2982us) negotiation.cc:107] Waiting for socket to connect
0928 20:21:10.487958 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 20:21:10.488068 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:21:10.488292 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:21:10.488293 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:21:10.488306 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:21:10.488391 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:21:10.488393 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:21:10.489456 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:21:10.489457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:21:10.491740 (+ 2283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:21:10.491744 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:21:10.492881 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:21:10.492883 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:21:10.492958 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:21:10.492963 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 20:21:10.493324 (+ 361us) client_negotiation.cc:240] Negotiation successful
0928 20:21:10.493445 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2959,"mutex_wait_us":48,"thread_start_us":110,"threads_started":1}
I20260928 20:21:10.493923 14795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f1dc70a42b43da9a431e2c63c15ece" candidate_uuid: "d9faf48b19064ab8a541c3febc28898c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95f0fbb445d148c597f20f823517cab2" is_pre_election: true
I20260928 20:21:10.494107 14795 raft_consensus.cc:2454] T 95f1dc70a42b43da9a431e2c63c15ece P 95f0fbb445d148c597f20f823517cab2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9faf48b19064ab8a541c3febc28898c in term 0.
I20260928 20:21:10.494338 14460 leader_election.cc:302] T 95f1dc70a42b43da9a431e2c63c15ece P d9faf48b19064ab8a541c3febc28898c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95f0fbb445d148c597f20f823517cab2, d9faf48b19064ab8a541c3febc28898c; no voters:
I20260928 20:21:10.494467 14867 raft_consensus.cc:2781] T 95f1dc70a42b43da9a431e2c63c15ece P d9faf48b19064ab8a541c3febc28898c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 20:21:10.494530 14867 raft_consensus.cc:482] T 95f1dc70a42b43da9a431e2c63c15ece P d9faf48b19064ab8a541c3febc28898c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 20:21:10.494557 14867 raft_consensus.cc:3037] T 95f1dc70a42b43da9a431e2c63c15ece P d9faf48b19064ab8a541c3febc28898c [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:21:10.495337 14867 raft_consensus.cc:504] T 95f1dc70a42b43da9a431e2c63c15ece P d9faf48b19064ab8a541c3febc28898c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e88c0e7b03e14d61b08ecda72422cd0e" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34261 } } peers { permanent_uuid: "d9faf48b19064ab8a541c3febc28898c" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 38105 } } peers { permanent_uuid: "95f0fbb445d148c597f20f823517cab2" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 39491 } }
I20260928 20:21:10.495520 14867 leader_election.cc:288] T 95f1dc70a42b43da9a431e2c63c15ece P d9faf48b19064ab8a541c3febc28898c [CANDIDATE]: Term 1 election: Requested vote from peers e88c0e7b03e14d61b08ecda72422cd0e (127.3.18.2:34261), 95f0fbb445d148c597f20f823517cab2 (127.3.18.3:39491)
I20260928 20:21:10.495837 14795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f1dc70a42b43da9a431e2c63c15ece" candidate_uuid: "d9faf48b19064ab8a541c3febc28898c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95f0fbb445d148c597f20f823517cab2"
I20260928 20:21:10.495937 14795 raft_consensus.cc:3037] T 95f1dc70a42b43da9a431e2c63c15ece P 95f0fbb445d148c597f20f823517cab2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:21:10.496963 14795 raft_consensus.cc:2454] T 95f1dc70a42b43da9a431e2c63c15ece P 95f0fbb445d148c597f20f823517cab2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9faf48b19064ab8a541c3febc28898c in term 1.
I20260928 20:21:10.497191 14860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:21:10.485548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:42917 (local address 127.3.18.2:34261)
0928 20:21:10.485590 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 20:21:10.485592 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:21:10.491103 (+ 5511us) server_negotiation.cc:407] Connection header received
0928 20:21:10.491537 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:21:10.491539 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:21:10.491555 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:21:10.491591 (+ 36us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:21:10.492142 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:21:10.492960 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:21:10.496074 (+ 3114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:21:10.496715 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:21:10.496774 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:21:10.496787 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0928 20:21:10.497051 (+ 264us) server_negotiation.cc:299] Negotiation successful
0928 20:21:10.497107 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 20:21:10.497241 14460 leader_election.cc:302] T 95f1dc70a42b43da9a431e2c63c15ece P d9faf48b19064ab8a541c3febc28898c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95f0fbb445d148c597f20f823517cab2, d9faf48b19064ab8a541c3febc28898c; no voters:
I20260928 20:21:10.497334 14867 raft_consensus.cc:2781] T 95f1dc70a42b43da9a431e2c63c15ece P d9faf48b19064ab8a541c3febc28898c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:21:10.497509 14867 raft_consensus.cc:686] T 95f1dc70a42b43da9a431e2c63c15ece P d9faf48b19064ab8a541c3febc28898c [term 1 LEADER]: Becoming Leader. State: Replica: d9faf48b19064ab8a541c3febc28898c, State: Running, Role: LEADER
I20260928 20:21:10.497620 14867 consensus_queue.cc:237] T 95f1dc70a42b43da9a431e2c63c15ece P d9faf48b19064ab8a541c3febc28898c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e88c0e7b03e14d61b08ecda72422cd0e" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34261 } } peers { permanent_uuid: "d9faf48b19064ab8a541c3febc28898c" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 38105 } } peers { permanent_uuid: "95f0fbb445d148c597f20f823517cab2" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 39491 } }
I20260928 20:21:10.498593 14872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:21:10.485370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.2:34261 (local address 127.3.18.1:42917)
0928 20:21:10.491005 (+ 5635us) negotiation.cc:107] Waiting for socket to connect
0928 20:21:10.491019 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 20:21:10.491231 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:21:10.491885 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:21:10.491886 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:21:10.491888 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:21:10.491963 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:21:10.491965 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:21:10.493185 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:21:10.493185 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:21:10.495964 (+ 2779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:21:10.495967 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:21:10.496859 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:21:10.496860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:21:10.496948 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:21:10.496954 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 20:21:10.498428 (+ 1474us) client_negotiation.cc:240] Negotiation successful
0928 20:21:10.498492 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5579,"thread_start_us":81,"threads_started":1}
I20260928 20:21:10.498615 14359 catalog_manager.cc:5495] T 95f1dc70a42b43da9a431e2c63c15ece P d9faf48b19064ab8a541c3febc28898c reported cstate change: term changed from 0 to 1, leader changed from <none> to d9faf48b19064ab8a541c3febc28898c (127.3.18.1). New cstate: current_term: 1 leader_uuid: "d9faf48b19064ab8a541c3febc28898c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e88c0e7b03e14d61b08ecda72422cd0e" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9faf48b19064ab8a541c3febc28898c" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 38105 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95f0fbb445d148c597f20f823517cab2" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 39491 } health_report { overall_health: UNKNOWN } } }
I20260928 20:21:10.499006 14664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f1dc70a42b43da9a431e2c63c15ece" candidate_uuid: "d9faf48b19064ab8a541c3febc28898c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e88c0e7b03e14d61b08ecda72422cd0e" is_pre_election: true
I20260928 20:21:10.499181 14664 raft_consensus.cc:2454] T 95f1dc70a42b43da9a431e2c63c15ece P e88c0e7b03e14d61b08ecda72422cd0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9faf48b19064ab8a541c3febc28898c in term 0.
I20260928 20:21:10.499328 14663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f1dc70a42b43da9a431e2c63c15ece" candidate_uuid: "d9faf48b19064ab8a541c3febc28898c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e88c0e7b03e14d61b08ecda72422cd0e"
I20260928 20:21:10.499425 14663 raft_consensus.cc:3037] T 95f1dc70a42b43da9a431e2c63c15ece P e88c0e7b03e14d61b08ecda72422cd0e [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:21:10.500394 14663 raft_consensus.cc:2454] T 95f1dc70a42b43da9a431e2c63c15ece P e88c0e7b03e14d61b08ecda72422cd0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9faf48b19064ab8a541c3febc28898c in term 1.
I20260928 20:21:10.535223 14861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:21:10.531688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52536 (local address 127.3.18.1:38105)
0928 20:21:10.531787 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0928 20:21:10.531789 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:21:10.531798 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:21:10.531840 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:21:10.531841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:21:10.531866 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:21:10.531949 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:21:10.532184 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:21:10.532938 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:21:10.534073 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:21:10.534369 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:21:10.534429 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:21:10.534626 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:21:10.534783 (+ 157us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:21:10.534859 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 20:21:10.535086 (+ 227us) server_negotiation.cc:299] Negotiation successful
0928 20:21:10.535108 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 20:21:10.537256 14795 raft_consensus.cc:1261] T 95f1dc70a42b43da9a431e2c63c15ece P 95f0fbb445d148c597f20f823517cab2 [term 1 FOLLOWER]: Refusing update from remote peer d9faf48b19064ab8a541c3febc28898c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:21:10.537318 14663 raft_consensus.cc:1261] T 95f1dc70a42b43da9a431e2c63c15ece P e88c0e7b03e14d61b08ecda72422cd0e [term 1 FOLLOWER]: Refusing update from remote peer d9faf48b19064ab8a541c3febc28898c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:21:10.537604 14873 consensus_queue.cc:1035] T 95f1dc70a42b43da9a431e2c63c15ece P d9faf48b19064ab8a541c3febc28898c [LEADER]: Connected to new peer: Peer: permanent_uuid: "95f0fbb445d148c597f20f823517cab2" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 39491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:21:10.537792 14867 consensus_queue.cc:1035] T 95f1dc70a42b43da9a431e2c63c15ece P d9faf48b19064ab8a541c3febc28898c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e88c0e7b03e14d61b08ecda72422cd0e" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:21:10.541051 14876 mvcc.cc:205] Tried to move back new op lower bound from 7334407661717848064 to 7334407661559197696. Current Snapshot: MvccSnapshot[applied={T|T < 7334407661717848064}]
I20260928 20:21:10.541927 14878 mvcc.cc:205] Tried to move back new op lower bound from 7334407661717848064 to 7334407661559197696. Current Snapshot: MvccSnapshot[applied={T|T < 7334407661717848064}]
I20260928 20:21:10.543004 14877 mvcc.cc:205] Tried to move back new op lower bound from 7334407661717848064 to 7334407661559197696. Current Snapshot: MvccSnapshot[applied={T|T < 7334407661717848064}]
I20260928 20:21:10.544450 14359 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:21:10.544550 14359 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:21:10.545419 14359 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1e0a55a0256147998ef65d1dc0c80250: Sending DeleteTablet for 3 replicas of tablet 95f1dc70a42b43da9a431e2c63c15ece
I20260928 20:21:10.545799 14644 tablet_service.cc:1514] Processing DeleteTablet for tablet 95f1dc70a42b43da9a431e2c63c15ece with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:21:10 UTC) from {username='kudu', principal='kudu/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:37762
I20260928 20:21:10.545854 14778 tablet_service.cc:1514] Processing DeleteTablet for tablet 95f1dc70a42b43da9a431e2c63c15ece with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:21:10 UTC) from {username='kudu', principal='kudu/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:53750
I20260928 20:21:10.546125 14506 tablet_service.cc:1514] Processing DeleteTablet for tablet 95f1dc70a42b43da9a431e2c63c15ece with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:21:10 UTC) from {username='kudu', principal='kudu/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:52526
I20260928 20:21:10.546705 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 14441
I20260928 20:21:10.548489 14887 tablet_replica.cc:330] T 95f1dc70a42b43da9a431e2c63c15ece P 95f0fbb445d148c597f20f823517cab2: stopping tablet replica
I20260928 20:21:10.548707 14887 raft_consensus.cc:2229] T 95f1dc70a42b43da9a431e2c63c15ece P 95f0fbb445d148c597f20f823517cab2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:21:10.548918 14887 raft_consensus.cc:2258] T 95f1dc70a42b43da9a431e2c63c15ece P 95f0fbb445d148c597f20f823517cab2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:21:10.549108 14886 tablet_replica.cc:330] T 95f1dc70a42b43da9a431e2c63c15ece P e88c0e7b03e14d61b08ecda72422cd0e: stopping tablet replica
I20260928 20:21:10.549257 14886 raft_consensus.cc:2229] T 95f1dc70a42b43da9a431e2c63c15ece P e88c0e7b03e14d61b08ecda72422cd0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:21:10.549387 14886 raft_consensus.cc:2258] T 95f1dc70a42b43da9a431e2c63c15ece P e88c0e7b03e14d61b08ecda72422cd0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:21:10.549921 14886 ts_tablet_manager.cc:1905] T 95f1dc70a42b43da9a431e2c63c15ece P e88c0e7b03e14d61b08ecda72422cd0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:21:10.551977 14886 ts_tablet_manager.cc:1918] T 95f1dc70a42b43da9a431e2c63c15ece P e88c0e7b03e14d61b08ecda72422cd0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:21:10.552124 14886 log.cc:1196] T 95f1dc70a42b43da9a431e2c63c15ece P e88c0e7b03e14d61b08ecda72422cd0e: Deleting WAL directory at /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790626764250411-3144-0/minicluster-data/ts-1/wal/wals/95f1dc70a42b43da9a431e2c63c15ece
I20260928 20:21:10.552497 14886 ts_tablet_manager.cc:1939] T 95f1dc70a42b43da9a431e2c63c15ece P e88c0e7b03e14d61b08ecda72422cd0e: Deleting consensus metadata
I20260928 20:21:10.553027 14346 catalog_manager.cc:4841] TS e88c0e7b03e14d61b08ecda72422cd0e (127.3.18.2:34261): tablet 95f1dc70a42b43da9a431e2c63c15ece (table test-table [id=9b4aebef86c64e91a8cc1882674a35b1]) successfully deleted
W20260928 20:21:10.554409 14344 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.18.1:38105 (error 108)
W20260928 20:21:10.555080 14857 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:21:10.554829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.1:38105 (local address 127.0.0.1:52548)
0928 20:21:10.554881 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 20:21:10.554892 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:21:10.554930 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:21:10.555033 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.1:38105: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 20:21:10.555205 14344 catalog_manager.cc:4584] TS d9faf48b19064ab8a541c3febc28898c (127.3.18.1:38105): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 95f1dc70a42b43da9a431e2c63c15ece: Network error: Client connection negotiation failed: client connection to 127.3.18.1:38105: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 20:21:10.555621 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 14578
I20260928 20:21:10.560031 14887 ts_tablet_manager.cc:1905] T 95f1dc70a42b43da9a431e2c63c15ece P 95f0fbb445d148c597f20f823517cab2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:21:10.562578 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 14715
I20260928 20:21:10.567518 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 14327
I20260928 20:21:10.573505 3144 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 20:21:10.909227 3144 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 20:21:10.909363 3144 mini_ranger.cc:66] Stopping Ranger...
I20260928 20:21:11.254439 3144 mini_ranger.cc:68] Stopped Ranger
2026-09-28 20:21:11.254 UTC [13913] LOG: received smart shutdown request
2026-09-28 20:21:11.258 UTC [13913] LOG: background worker "logical replication launcher" (PID 13922) exited with exit code 1
2026-09-28 20:21:11.259 UTC [13917] LOG: shutting down
2026-09-28 20:21:11.478 UTC [13913] LOG: database system is shut down
2026-09-28T20:21:11Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (51375 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:21:11 dist-test-slave-b83v krb5kdc[14898](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:21:11 dist-test-slave-b83v krb5kdc[14898](info): set up 2 sockets
Sep 28 20:21:11 dist-test-slave-b83v krb5kdc[14898](info): commencing operation
krb5kdc: starting...
W20260928 20:21:12.644042 3144 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Sep 28 20:21:12 dist-test-slave-b83v krb5kdc[14898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:21:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:21:12Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.3.18.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.3.18.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.3.18.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/krb5kdc/rangeradmin_127.3.18.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.3.18.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/krb5kdc/rangeradmin_127.3.18.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.3.18.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.3.18.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.3.18.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/krb5kdc/rangerlookup_127.3.18.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.3.18.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/krb5kdc/rangerlookup_127.3.18.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.18.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.18.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.18.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/krb5kdc/HTTP_127.3.18.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.18.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/krb5kdc/HTTP_127.3.18.20@KRBTEST.COM.keytab.
I20260928 20:21:12.828585 3144 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task2yfqO7/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/postgres -l logfile start
2026-09-28 20:21:15.682 UTC [14928] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 20:21:15.682 UTC [14928] LOG: listening on IPv4 address "127.3.18.20", port 43217
2026-09-28 20:21:15.687 UTC [14928] LOG: listening on Unix socket "/tmp/.s.PGSQL.43217"
2026-09-28 20:21:15.724 UTC [14931] LOG: database system was shut down at 2026-09-28 20:21:13 UTC
2026-09-28 20:21:15.729 UTC [14928] LOG: database system is ready to accept connections
I20260928 20:21:16.691960 3144 mini_postgres.cc:96] Postgres bound to 43217
2026-09-28 20:21:16.698 UTC [14939] FATAL: database "slave" does not exist
127.3.18.20:43217 - accepting connections
I20260928 20:21:16.698844 3144 mini_ranger.cc:161] Starting Ranger...
I20260928 20:21:16.714397 3144 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 20:21:17.331079 3144 mini_ranger.cc:87] Created ranger Postgres database
I20260928 20:21:17.331192 3144 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ranger-admin
2026-09-28 20:21:17,621 [I] DB FLAVOR :POSTGRES
2026-09-28 20:21:17,622 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 20:21:17,622 [I] Checking connection..
2026-09-28 20:21:17,622 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2yfqO7/build/release/bin/postgresql.jar:/tmp/dist-test-task2yfqO7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.18.20:43217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 20:21:17,901 [I] Checking connection passed.
2026-09-28 20:21:17,902 [I] --------- Verifying version history table ---------
2026-09-28 20:21:17,902 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2yfqO7/build/release/bin/postgresql.jar:/tmp/dist-test-task2yfqO7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.18.20:43217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 20:21:18,177 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 20:21:18,177 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2yfqO7/build/release/bin/postgresql.jar:/tmp/dist-test-task2yfqO7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.18.20:43217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 20:21:18,425 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 20:21:18,425 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 20:21:18,426 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2yfqO7/build/release/bin/postgresql.jar:/tmp/dist-test-task2yfqO7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.18.20:43217/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task2yfqO7/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 20:21:18.696 UTC [15003] WARNING: there is no transaction in progress
2026-09-28 20:21:18,703 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 20:21:18,703 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2yfqO7/build/release/bin/postgresql.jar:/tmp/dist-test-task2yfqO7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.18.20:43217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 20:21:18,978 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 20:21:18,979 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 20:21:18,979 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2yfqO7/build/release/bin/postgresql.jar:/tmp/dist-test-task2yfqO7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.18.20:43217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 20:21:19,242 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2yfqO7/build/release/bin/postgresql.jar:/tmp/dist-test-task2yfqO7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.18.20:43217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 20:21:19,493 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2yfqO7/build/release/bin/postgresql.jar:/tmp/dist-test-task2yfqO7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.18.20:43217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-b83v.c.gcp-upstream.internal','N') ;"
2026-09-28 20:21:19,731 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2yfqO7/build/release/bin/postgresql.jar:/tmp/dist-test-task2yfqO7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.18.20:43217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 20:21:20,002 [I] Table x_portal_user does not exist in database ranger
2026-09-28 20:21:20,003 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2yfqO7/build/release/bin/postgresql.jar:/tmp/dist-test-task2yfqO7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.18.20:43217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 20:21:20,274 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 20:21:20,274 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2yfqO7/build/release/bin/postgresql.jar:/tmp/dist-test-task2yfqO7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.18.20:43217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 20:21:20,544 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2yfqO7/build/release/bin/postgresql.jar:/tmp/dist-test-task2yfqO7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.18.20:43217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 20:21:20,805 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 20:21:20,805 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2yfqO7/build/release/bin/postgresql.jar:/tmp/dist-test-task2yfqO7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.18.20:43217/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task2yfqO7/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 20:21:21.941 UTC [15111] WARNING: there is no transaction in progress
2026-09-28 20:21:21.962 UTC [15111] WARNING: there is no transaction in progress
2026-09-28 20:21:21.981 UTC [15111] WARNING: there is no transaction in progress
2026-09-28 20:21:21.998 UTC [15111] WARNING: there is no transaction in progress
2026-09-28 20:21:22.016 UTC [15111] WARNING: there is no transaction in progress
2026-09-28 20:21:22.035 UTC [15111] WARNING: there is no transaction in progress
2026-09-28 20:21:22.099 UTC [15111] WARNING: there is no transaction in progress
2026-09-28 20:21:22.107 UTC [15111] WARNING: there is no transaction in progress
2026-09-28 20:21:22.123 UTC [15111] WARNING: there is no transaction in progress
2026-09-28 20:21:22.135 UTC [15111] WARNING: there is no transaction in progress
2026-09-28 20:21:22.150 UTC [15111] WARNING: there is no transaction in progress
2026-09-28 20:21:22.168 UTC [15111] WARNING: there is no transaction in progress
2026-09-28 20:21:22.181 UTC [15111] WARNING: there is no transaction in progress
2026-09-28 20:21:22.192 UTC [15111] WARNING: there is no transaction in progress
2026-09-28 20:21:22.202 UTC [15111] WARNING: there is no transaction in progress
2026-09-28 20:21:22.212 UTC [15111] WARNING: there is no transaction in progress
2026-09-28 20:21:23,054 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 20:21:23,054 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2yfqO7/build/release/bin/postgresql.jar:/tmp/dist-test-task2yfqO7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.18.20:43217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 20:21:23,302 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 20:21:23,303 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2yfqO7/build/release/bin/postgresql.jar:/tmp/dist-test-task2yfqO7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.18.20:43217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 20:21:23,552 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 20:21:23,553 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2yfqO7/build/release/bin/postgresql.jar:/tmp/dist-test-task2yfqO7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.18.20:43217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 20:21:23,798 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 20:21:23,799 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2yfqO7/build/release/bin/postgresql.jar:/tmp/dist-test-task2yfqO7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.18.20:43217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 20:21:24,067 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2yfqO7/build/release/bin/postgresql.jar:/tmp/dist-test-task2yfqO7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.18.20:43217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 20:21:24,349 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2yfqO7/build/release/bin/postgresql.jar:/tmp/dist-test-task2yfqO7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.18.20:43217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-b83v.c.gcp-upstream.internal';"
2026-09-28 20:21:24,577 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 20:21:24,578 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2yfqO7/build/release/bin/postgresql.jar:/tmp/dist-test-task2yfqO7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.18.20:43217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 20:21:24,855 [I] DB_PATCHES have already been applied
I20260928 20:21:24.861083 3144 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ranger-admin:/tmp/dist-test-task2yfqO7/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task2yfqO7/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task2yfqO7/build/release/bin/postgresql.jar:/tmp/dist-test-task2yfqO7/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 20:21:24.861212 3144 mini_ranger.cc:193] Using host: 127.3.18.20
I20260928 20:21:24.864512 3144 mini_ranger.cc:227] Ranger admin URL: http://127.3.18.20:54921
Sep 28, 2026 8:21:25 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 8:21:25 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 8:21:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 8:21:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 8:21:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 8:21:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 8:21:25 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 20:21:25.978024 3144 mini_ranger.cc:160] Time spent starting Ranger: real 9.279s user 0.000s sys 0.009s
Sep 28, 2026 8:21:47 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 8:21:47 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 8:21:47 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 8:21:47 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 8:21:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 8:21:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 8:21:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 8:21:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 8:21:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 8:21:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 8:21:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 8:21:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 8:21:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 8:21:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 8:21:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 8:21:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 8:21:47 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 8:21:48 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 20:21:49.314589 3144 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.3.18.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.3.18.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.3.18.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/krb5kdc/rangerkms_127.3.18.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.3.18.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/krb5kdc/rangerkms_127.3.18.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.18.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.3.18.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.18.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/krb5kdc/HTTP_127.3.18.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.18.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/krb5kdc/HTTP_127.3.18.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 20:21:49 dist-test-slave-b83v krb5kdc[14898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626909, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 20:21:49.410979 3144 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 20:21:49.420665 3144 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 20:21:50.490628 3144 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 20:21:50.490736 3144 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ranger-kms
I20260928 20:21:50.490762 3144 mini_ranger_kms.cc:227] Using postgres at 127.3.18.20:43217
2026-09-28 20:21:50,664 [I] DB FLAVOR :POSTGRES
2026-09-28 20:21:50,664 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 20:21:50,665 [I] Checking connection
2026-09-28 20:21:50,665 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2yfqO7/build/release/bin/postgresql.jar:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.18.20:43217/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 20:21:50,926 [I] connection success
2026-09-28 20:21:50,927 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 20:21:50,927 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 20:21:50,927 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2yfqO7/build/release/bin/postgresql.jar:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.18.20:43217/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 20:21:51,187 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 20:21:51,188 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 20:21:51,188 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 20:21:51,188 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2yfqO7/build/release/bin/postgresql.jar:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.18.20:43217/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 20:21:51,506 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 20:21:52.084930 3144 mini_ranger_kms.cc:310] Created kms service
I20260928 20:21:52.297194 3144 mini_ranger_kms.cc:326] Created kudu user
I20260928 20:21:52.344777 3144 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 20:21:52 dist-test-slave-b83v krb5kdc[14898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626909, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.3.18.20@KRBTEST.COM
I20260928 20:21:52.810177 3144 mini_ranger_kms.cc:384] Added ranger policy
I20260928 20:21:52.810672 3144 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ranger-kms:/tmp/dist-test-task2yfqO7/build/release/bin/postgresql.jar:/tmp/dist-test-task2yfqO7/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task2yfqO7/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 20:21:52.810737 3144 mini_ranger_kms.cc:240] Using host: 127.3.18.20
I20260928 20:21:52.814317 3144 mini_ranger_kms.cc:276] Ranger KMS PID: 15292
I20260928 20:21:52.814416 3144 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.3.18.20:45363
Sep 28, 2026 8:21:53 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 8:21:53 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 8:21:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 8:21:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 8:21:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 8:21:54 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 20:21:55.111904 3144 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.701s user 0.001s sys 0.156s
I20260928 20:21:55.111994 3144 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 20:21:55.112051 3144 mini_ranger_kms.cc:401] 127.3.18.20:45363/kms/v1/keys
Sep 28 20:21:55 dist-test-slave-b83v krb5kdc[14898](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790626915, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.18.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 20:21:55 dist-test-slave-b83v krb5kdc[14898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626915, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.18.20@KRBTEST.COM for HTTP/127.3.18.20@KRBTEST.COM
Sep 28 20:21:56 dist-test-slave-b83v krb5kdc[14898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626915, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.18.20@KRBTEST.COM for HTTP/127.3.18.20@KRBTEST.COM
Sep 28 20:21:58 dist-test-slave-b83v krb5kdc[14898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.3.18.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab.
Entry for principal kudu/127.3.18.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab.
I20260928 20:21:58.427215 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:39369
--webserver_interface=127.3.18.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:44943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.18.62:39369
--ranger_config_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.18.20:45363/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:21:58.519939 15342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:21:58.520195 15342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:21:58.520219 15342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:21:58.521525 15342 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:21:58.521584 15342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:21:58.521600 15342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:21:58.521612 15342 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:21:58.521624 15342 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:21:58.523535 15342 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:44943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/master-0/wal
--ranger_kms_url=127.3.18.20:45363/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.18.62:39369
--ranger_config_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:39369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.18.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:21:58.523864 15342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:21:58.524183 15342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 20:21:58.530073 15342 server_base.cc:1034] running on GCE node
W20260928 20:21:58.530092 15348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:21:58.530090 15350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:21:58.530073 15347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:21:58.530524 15342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:21:58.530817 15342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:21:58.531958 15342 hybrid_clock.cc:648] HybridClock initialized: now 1790626918531949 us; error 36 us; skew 500 ppm
Sep 28 20:21:58 dist-test-slave-b83v krb5kdc[14898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626918, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:21:58.546095 15342 init.cc:377] Logged in from keytab as kudu/127.3.18.62@KRBTEST.COM (short username kudu)
I20260928 20:21:58.546581 15342 webserver.cc:460] Webserver started at http://127.3.18.62:34983/ using document root <none> and password file <none>
I20260928 20:21:58.546877 15342 fs_manager.cc:359] Metadata directory not provided
I20260928 20:21:58.546924 15342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:21:58.547065 15342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 20:21:58 dist-test-slave-b83v krb5kdc[14898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626918, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.62@KRBTEST.COM for HTTP/127.3.18.20@KRBTEST.COM
I20260928 20:21:58.603634 15342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/master-0/data/instance:
uuid: "7f803e5c20bf4ca781ffa4741ee68854"
format_stamp: "Formatted at 2026-09-28 20:21:58 on dist-test-slave-b83v"
server_key: "2c0c7c3c99db083629c90bf86329ca2f"
server_key_iv: "5e39a4781693fb17aaa66f86167ae72f"
server_key_version: "kuduclusterkey@0"
I20260928 20:21:58.604192 15342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/master-0/wal/instance:
uuid: "7f803e5c20bf4ca781ffa4741ee68854"
format_stamp: "Formatted at 2026-09-28 20:21:58 on dist-test-slave-b83v"
server_key: "2c0c7c3c99db083629c90bf86329ca2f"
server_key_iv: "5e39a4781693fb17aaa66f86167ae72f"
server_key_version: "kuduclusterkey@0"
I20260928 20:21:58.605856 15342 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 20:21:58.618194 15357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:21:58.618454 15342 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:21:58.618566 15342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/master-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/master-0/wal
uuid: "7f803e5c20bf4ca781ffa4741ee68854"
format_stamp: "Formatted at 2026-09-28 20:21:58 on dist-test-slave-b83v"
server_key: "2c0c7c3c99db083629c90bf86329ca2f"
server_key_iv: "5e39a4781693fb17aaa66f86167ae72f"
server_key_version: "kuduclusterkey@0"
I20260928 20:21:58.618669 15342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:21:58.634071 15342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:21:58.636238 15342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:21:58.636411 15342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:21:58.641430 15342 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.62:39369
I20260928 20:21:58.641453 15409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.62:39369 every 8 connection(s)
I20260928 20:21:58.641904 15342 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
I20260928 20:21:58.642822 15410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:21:58.644280 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 15342
I20260928 20:21:58.644429 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/master-0/wal/instance
I20260928 20:21:58.644629 3144 external_mini_cluster.cc:1428] Setting key 06265616b3f1221c03e321d24903e005
I20260928 20:21:58.646610 15410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f803e5c20bf4ca781ffa4741ee68854: Bootstrap starting.
I20260928 20:21:58.647466 15410 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f803e5c20bf4ca781ffa4741ee68854: Neither blocks nor log segments found. Creating new log.
I20260928 20:21:58.647891 15410 log.cc:824] T 00000000000000000000000000000000 P 7f803e5c20bf4ca781ffa4741ee68854: Log is configured to *not* fsync() on all Append() calls
I20260928 20:21:58.648811 15410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f803e5c20bf4ca781ffa4741ee68854: No bootstrap required, opened a new log
Sep 28 20:21:58 dist-test-slave-b83v krb5kdc[14898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:21:58.651656 15410 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f803e5c20bf4ca781ffa4741ee68854 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f803e5c20bf4ca781ffa4741ee68854" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 39369 } }
I20260928 20:21:58.651894 15410 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f803e5c20bf4ca781ffa4741ee68854 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:21:58.652001 15410 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f803e5c20bf4ca781ffa4741ee68854 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f803e5c20bf4ca781ffa4741ee68854, State: Initialized, Role: FOLLOWER
I20260928 20:21:58.652165 15410 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f803e5c20bf4ca781ffa4741ee68854 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f803e5c20bf4ca781ffa4741ee68854" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 39369 } }
I20260928 20:21:58.652297 15410 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f803e5c20bf4ca781ffa4741ee68854 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:21:58.652357 15410 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f803e5c20bf4ca781ffa4741ee68854 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:21:58.652421 15410 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f803e5c20bf4ca781ffa4741ee68854 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:21:58.653261 15410 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f803e5c20bf4ca781ffa4741ee68854 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f803e5c20bf4ca781ffa4741ee68854" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 39369 } }
I20260928 20:21:58.653434 15410 leader_election.cc:302] T 00000000000000000000000000000000 P 7f803e5c20bf4ca781ffa4741ee68854 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f803e5c20bf4ca781ffa4741ee68854; no voters:
I20260928 20:21:58.653661 15410 leader_election.cc:288] T 00000000000000000000000000000000 P 7f803e5c20bf4ca781ffa4741ee68854 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:21:58.653717 15415 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f803e5c20bf4ca781ffa4741ee68854 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:21:58.653996 15415 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f803e5c20bf4ca781ffa4741ee68854 [term 1 LEADER]: Becoming Leader. State: Replica: 7f803e5c20bf4ca781ffa4741ee68854, State: Running, Role: LEADER
I20260928 20:21:58.654057 15410 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f803e5c20bf4ca781ffa4741ee68854 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:21:58.654183 15415 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f803e5c20bf4ca781ffa4741ee68854 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f803e5c20bf4ca781ffa4741ee68854" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 39369 } }
I20260928 20:21:58.654544 15415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f803e5c20bf4ca781ffa4741ee68854 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f803e5c20bf4ca781ffa4741ee68854" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f803e5c20bf4ca781ffa4741ee68854" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 39369 } } }
I20260928 20:21:58.654619 15415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f803e5c20bf4ca781ffa4741ee68854 [sys.catalog]: This master's current role is: LEADER
I20260928 20:21:58.654742 15415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f803e5c20bf4ca781ffa4741ee68854 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f803e5c20bf4ca781ffa4741ee68854. Latest consensus state: current_term: 1 leader_uuid: "7f803e5c20bf4ca781ffa4741ee68854" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f803e5c20bf4ca781ffa4741ee68854" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 39369 } } }
I20260928 20:21:58.654784 15415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f803e5c20bf4ca781ffa4741ee68854 [sys.catalog]: This master's current role is: LEADER
I20260928 20:21:58.655300 15419 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:21:58.655867 15419 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:21:58.656701 15413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:21:58.645220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52078 (local address 127.3.18.62:39369)
0928 20:21:58.645451 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 20:21:58.645462 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 20:21:58.645500 (+ 38us) server_negotiation.cc:407] Connection header received
0928 20:21:58.646151 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:21:58.646160 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:21:58.646224 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:21:58.646424 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:21:58.647191 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:21:58.648418 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:21:58.649267 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:21:58.649696 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:21:58.649817 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:21:58.652280 (+ 2463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:21:58.652286 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:21:58.652286 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:21:58.652301 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:21:58.654409 (+ 2108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:21:58.654882 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:21:58.654886 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:21:58.654887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:21:58.654996 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:21:58.655221 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:21:58.655222 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:21:58.655223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:21:58.655572 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:21:58.655741 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 20:21:58.656344 (+ 603us) server_negotiation.cc:299] Negotiation successful
0928 20:21:58.656473 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":90,"threads_started":1}
I20260928 20:21:58.656769 15410 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 20:21:58.662150 15419 catalog_manager.cc:1347] Generated new cluster ID: 6eb8a040d45e4c249513957f91ab8130
I20260928 20:21:58.662317 15419 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:21:58.697897 15419 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:21:58.698822 15419 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:21:58.709930 15419 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f803e5c20bf4ca781ffa4741ee68854: Generated new TSK 0
I20260928 20:21:58.710187 15419 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.18.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab.
Entry for principal kudu/127.3.18.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab.
I20260928 20:21:59.476203 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.18.20:45363/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.1:0
--local_ip_for_outbound_sockets=127.3.18.1
--webserver_interface=127.3.18.1
--webserver_port=0
--tserver_master_addrs=127.3.18.62:39369
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab
--principal=kudu/127.3.18.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:44943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:21:59.572055 15455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:21:59.572288 15455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:21:59.572335 15455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:21:59.573493 15455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:21:59.573560 15455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:21:59.573603 15455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.1
I20260928 20:21:59.575129 15455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:44943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.3.18.20:45363/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab
--principal=kudu/127.3.18.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.18.1
--webserver_port=0
--tserver_master_addrs=127.3.18.62:39369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.1
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:21:59.575457 15455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:21:59.575742 15455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:21:59.581158 15461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:21:59.581378 15455 server_base.cc:1034] running on GCE node
W20260928 20:21:59.581173 15460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:21:59.581527 15463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:21:59.581822 15455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:21:59.582108 15455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:21:59.583279 15455 hybrid_clock.cc:648] HybridClock initialized: now 1790626919583255 us; error 40 us; skew 500 ppm
Sep 28 20:21:59 dist-test-slave-b83v krb5kdc[14898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626919, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:21:59.586110 15455 init.cc:377] Logged in from keytab as kudu/127.3.18.1@KRBTEST.COM (short username kudu)
I20260928 20:21:59.586653 15455 webserver.cc:460] Webserver started at http://127.3.18.1:40183/ using document root <none> and password file <none>
I20260928 20:21:59.586916 15455 fs_manager.cc:359] Metadata directory not provided
I20260928 20:21:59.586963 15455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:21:59.587105 15455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 20:21:59 dist-test-slave-b83v krb5kdc[14898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626919, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.1@KRBTEST.COM for HTTP/127.3.18.20@KRBTEST.COM
I20260928 20:21:59.600853 15455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-0/data/instance:
uuid: "114f7db88e624bf5b08061aad81886a0"
format_stamp: "Formatted at 2026-09-28 20:21:59 on dist-test-slave-b83v"
server_key: "f44741b46fa4d51907bfbe2a1e30d594"
server_key_iv: "475d649207abc5f1319f944c8161e673"
server_key_version: "kuduclusterkey@0"
I20260928 20:21:59.601408 15455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-0/wal/instance:
uuid: "114f7db88e624bf5b08061aad81886a0"
format_stamp: "Formatted at 2026-09-28 20:21:59 on dist-test-slave-b83v"
server_key: "f44741b46fa4d51907bfbe2a1e30d594"
server_key_iv: "475d649207abc5f1319f944c8161e673"
server_key_version: "kuduclusterkey@0"
I20260928 20:21:59.603452 15455 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260928 20:21:59.621040 15470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:21:59.621407 15455 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260928 20:21:59.621614 15455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-0/wal
uuid: "114f7db88e624bf5b08061aad81886a0"
format_stamp: "Formatted at 2026-09-28 20:21:59 on dist-test-slave-b83v"
server_key: "f44741b46fa4d51907bfbe2a1e30d594"
server_key_iv: "475d649207abc5f1319f944c8161e673"
server_key_version: "kuduclusterkey@0"
I20260928 20:21:59.621737 15455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:21:59.638639 15455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:21:59.641124 15455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:21:59.641430 15455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:21:59.641746 15455 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:21:59.642225 15455 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:21:59.642313 15455 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:21:59.642364 15455 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:21:59.642382 15455 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:21:59.649955 15455 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.1:41821
I20260928 20:21:59.650040 15583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.1:41821 every 8 connection(s)
I20260928 20:21:59.650364 15455 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
I20260928 20:21:59.652568 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 15455
I20260928 20:21:59.652684 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-0/wal/instance
I20260928 20:21:59.652890 3144 external_mini_cluster.cc:1428] Setting key de6d6b9e458eff332d959400341affbe
Sep 28 20:21:59 dist-test-slave-b83v krb5kdc[14898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626919, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.1@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.18.2@KRBTEST.COM; defaulting to no policy
I20260928 20:21:59.669049 15587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:21:59.651175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:37607 (local address 127.3.18.62:39369)
0928 20:21:59.651327 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 20:21:59.651330 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:21:59.651488 (+ 158us) server_negotiation.cc:407] Connection header received
0928 20:21:59.652246 (+ 758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:21:59.652247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:21:59.652263 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:21:59.652356 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:21:59.654234 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:21:59.655068 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:21:59.656762 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:21:59.657119 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:21:59.657195 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:21:59.664412 (+ 7217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:21:59.664414 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:21:59.664415 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:21:59.664426 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:21:59.666865 (+ 2439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:21:59.667573 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:21:59.667575 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:21:59.667575 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:21:59.667655 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:21:59.668084 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:21:59.668086 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:21:59.668086 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:21:59.668253 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:21:59.668352 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 20:21:59.668800 (+ 448us) server_negotiation.cc:299] Negotiation successful
0928 20:21:59.668919 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":75,"threads_started":1}
I20260928 20:21:59.669256 15586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:21:59.651094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:39369 (local address 127.3.18.1:37607)
0928 20:21:59.651310 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 20:21:59.651361 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 20:21:59.652099 (+ 738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:21:59.653187 (+ 1088us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:21:59.653193 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:21:59.653469 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:21:59.653800 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:21:59.653808 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:21:59.655187 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:21:59.655188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:21:59.656669 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:21:59.656673 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:21:59.658288 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:21:59.658289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:21:59.658472 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:21:59.659614 (+ 1142us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:21:59.659627 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:21:59.664222 (+ 4595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:21:59.667026 (+ 2804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:21:59.667042 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:21:59.667046 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:21:59.667450 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:21:59.667736 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:21:59.667737 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:21:59.667738 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:21:59.667958 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:21:59.668343 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:21:59.668354 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:21:59.668620 (+ 266us) client_negotiation.cc:769] Sending connection context
0928 20:21:59.668761 (+ 141us) client_negotiation.cc:240] Negotiation successful
0928 20:21:59.668962 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":95,"threads_started":1}
I20260928 20:21:59.669876 15584 heartbeater.cc:347] Connected to a master server at 127.3.18.62:39369
I20260928 20:21:59.669993 15584 heartbeater.cc:464] Registering TS with master...
I20260928 20:21:59.670218 15584 heartbeater.cc:511] Master 127.3.18.62:39369 requested a full tablet report, sending...
I20260928 20:21:59.670806 15374 ts_manager.cc:194] Registered new tserver with Master: 114f7db88e624bf5b08061aad81886a0 (127.3.18.1:41821)
I20260928 20:21:59.671463 15374 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.18.1@KRBTEST.COM'} at 127.3.18.1:37607
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab.
Entry for principal kudu/127.3.18.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab.
I20260928 20:21:59.691121 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.18.20:45363/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.2:0
--local_ip_for_outbound_sockets=127.3.18.2
--webserver_interface=127.3.18.2
--webserver_port=0
--tserver_master_addrs=127.3.18.62:39369
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab
--principal=kudu/127.3.18.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:44943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:21:59.799731 15592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:21:59.800055 15592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:21:59.800097 15592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:21:59.801337 15592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:21:59.801409 15592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:21:59.801452 15592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.2
I20260928 20:21:59.803031 15592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:44943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.3.18.20:45363/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab
--principal=kudu/127.3.18.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.18.2
--webserver_port=0
--tserver_master_addrs=127.3.18.62:39369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.2
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:21:59.803323 15592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:21:59.803577 15592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:21:59.808740 15602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:21:59.808874 15600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:21:59.809249 15599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:21:59.809378 15592 server_base.cc:1034] running on GCE node
I20260928 20:21:59.809650 15592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:21:59.809969 15592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:21:59.811137 15592 hybrid_clock.cc:648] HybridClock initialized: now 1790626919811119 us; error 37 us; skew 500 ppm
Sep 28 20:21:59 dist-test-slave-b83v krb5kdc[14898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626919, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:21:59.813920 15592 init.cc:377] Logged in from keytab as kudu/127.3.18.2@KRBTEST.COM (short username kudu)
I20260928 20:21:59.814446 15592 webserver.cc:460] Webserver started at http://127.3.18.2:36077/ using document root <none> and password file <none>
I20260928 20:21:59.814695 15592 fs_manager.cc:359] Metadata directory not provided
I20260928 20:21:59.814765 15592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:21:59.814908 15592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 20:21:59 dist-test-slave-b83v krb5kdc[14898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626919, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.2@KRBTEST.COM for HTTP/127.3.18.20@KRBTEST.COM
I20260928 20:21:59.828473 15592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-1/data/instance:
uuid: "1c75cca332aa4d3db5010436100997ae"
format_stamp: "Formatted at 2026-09-28 20:21:59 on dist-test-slave-b83v"
server_key: "7938386a3f04dfcd175449d2d910cb31"
server_key_iv: "cc9cde8aa3ec413a7be1d9946ae5bf57"
server_key_version: "kuduclusterkey@0"
I20260928 20:21:59.829087 15592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-1/wal/instance:
uuid: "1c75cca332aa4d3db5010436100997ae"
format_stamp: "Formatted at 2026-09-28 20:21:59 on dist-test-slave-b83v"
server_key: "7938386a3f04dfcd175449d2d910cb31"
server_key_iv: "cc9cde8aa3ec413a7be1d9946ae5bf57"
server_key_version: "kuduclusterkey@0"
I20260928 20:21:59.830758 15592 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:21:59.847329 15609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:21:59.847554 15592 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:21:59.847661 15592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-1/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-1/wal
uuid: "1c75cca332aa4d3db5010436100997ae"
format_stamp: "Formatted at 2026-09-28 20:21:59 on dist-test-slave-b83v"
server_key: "7938386a3f04dfcd175449d2d910cb31"
server_key_iv: "cc9cde8aa3ec413a7be1d9946ae5bf57"
server_key_version: "kuduclusterkey@0"
I20260928 20:21:59.847800 15592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 28 20:21:59 dist-test-slave-b83v krb5kdc[14898](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790626919, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:21:59.868628 15592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:21:59.871279 15592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:21:59.871497 15592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:21:59.871834 15592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:21:59.872311 15592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:21:59.872445 15592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:21:59.872545 15592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:21:59.872603 15592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:21:59.879015 15592 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.2:34165
I20260928 20:21:59.879560 15592 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
I20260928 20:21:59.884094 15722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.2:34165 every 8 connection(s)
I20260928 20:21:59.888414 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 15592
I20260928 20:21:59.888530 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-1/wal/instance
I20260928 20:21:59.888727 3144 external_mini_cluster.cc:1428] Setting key 53121240152ef5e73d7e63f8f33ae11b
Sep 28 20:21:59 dist-test-slave-b83v krb5kdc[14898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626919, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.2@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:21:59.896499 15587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:21:59.885438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:40473 (local address 127.3.18.62:39369)
0928 20:21:59.885514 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 20:21:59.885517 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:21:59.885675 (+ 158us) server_negotiation.cc:407] Connection header received
0928 20:21:59.886434 (+ 759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:21:59.886436 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:21:59.886449 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:21:59.886483 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:21:59.887325 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:21:59.887980 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:21:59.889858 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:21:59.890179 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:21:59.890221 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:21:59.892736 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:21:59.892737 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:21:59.892738 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:21:59.892746 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:21:59.894604 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:21:59.895170 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:21:59.895171 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:21:59.895172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:21:59.895232 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:21:59.895526 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:21:59.895527 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:21:59.895527 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:21:59.895689 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:21:59.895737 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 20:21:59.896255 (+ 518us) server_negotiation.cc:299] Negotiation successful
0928 20:21:59.896364 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 20:21:59.897423 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:21:59.885139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:39369 (local address 127.3.18.2:40473)
0928 20:21:59.885544 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0928 20:21:59.885591 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 20:21:59.886290 (+ 699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:21:59.886683 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:21:59.886688 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:21:59.886903 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:21:59.887211 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:21:59.887217 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:21:59.888096 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:21:59.888096 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:21:59.889770 (+ 1674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:21:59.889773 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:21:59.890321 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:21:59.890322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:21:59.890484 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:21:59.891121 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:21:59.891131 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:21:59.892638 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:21:59.894708 (+ 2070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:21:59.894718 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:21:59.894721 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:21:59.895071 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:21:59.895304 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:21:59.895305 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:21:59.895305 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:21:59.895445 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:21:59.895842 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:21:59.895850 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:21:59.896074 (+ 224us) client_negotiation.cc:769] Sending connection context
0928 20:21:59.896942 (+ 868us) client_negotiation.cc:240] Negotiation successful
0928 20:21:59.897142 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":84,"threads_started":1}
I20260928 20:21:59.898193 15723 heartbeater.cc:347] Connected to a master server at 127.3.18.62:39369
I20260928 20:21:59.898294 15723 heartbeater.cc:464] Registering TS with master...
I20260928 20:21:59.898569 15723 heartbeater.cc:511] Master 127.3.18.62:39369 requested a full tablet report, sending...
I20260928 20:21:59.898981 15374 ts_manager.cc:194] Registered new tserver with Master: 1c75cca332aa4d3db5010436100997ae (127.3.18.2:34165)
I20260928 20:21:59.899520 15374 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.18.2@KRBTEST.COM'} at 127.3.18.2:40473
WARNING: no policy specified for kudu/127.3.18.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab.
Entry for principal kudu/127.3.18.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab.
I20260928 20:21:59.924681 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.18.20:45363/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.3:0
--local_ip_for_outbound_sockets=127.3.18.3
--webserver_interface=127.3.18.3
--webserver_port=0
--tserver_master_addrs=127.3.18.62:39369
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab
--principal=kudu/127.3.18.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:44943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:22:00.020011 15728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:22:00.020387 15728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:22:00.020486 15728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:22:00.022406 15728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:22:00.022519 15728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:22:00.022614 15728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.3
I20260928 20:22:00.024544 15728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:44943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.3.18.20:45363/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab
--principal=kudu/127.3.18.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.18.3
--webserver_port=0
--tserver_master_addrs=127.3.18.62:39369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.3
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:22:00.024785 15728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:22:00.025012 15728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:22:00.030450 15734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:22:00.030661 15728 server_base.cc:1034] running on GCE node
W20260928 20:22:00.030864 15735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:22:00.030457 15737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:22:00.031247 15728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:22:00.031715 15728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:22:00.033036 15728 hybrid_clock.cc:648] HybridClock initialized: now 1790626920033011 us; error 40 us; skew 500 ppm
Sep 28 20:22:00 dist-test-slave-b83v krb5kdc[14898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626920, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:22:00.036514 15728 init.cc:377] Logged in from keytab as kudu/127.3.18.3@KRBTEST.COM (short username kudu)
I20260928 20:22:00.037133 15728 webserver.cc:460] Webserver started at http://127.3.18.3:44227/ using document root <none> and password file <none>
I20260928 20:22:00.037425 15728 fs_manager.cc:359] Metadata directory not provided
I20260928 20:22:00.037510 15728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:22:00.037717 15728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 20:22:00 dist-test-slave-b83v krb5kdc[14898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626920, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.3@KRBTEST.COM for HTTP/127.3.18.20@KRBTEST.COM
I20260928 20:22:00.057111 15728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-2/data/instance:
uuid: "a66b3f431af44105ac0df4a076b15063"
format_stamp: "Formatted at 2026-09-28 20:22:00 on dist-test-slave-b83v"
server_key: "d789dfb5977c30c18014e465026ad1d0"
server_key_iv: "ab8b866b3605f61899cbbad1463e04d4"
server_key_version: "kuduclusterkey@0"
I20260928 20:22:00.057631 15728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-2/wal/instance:
uuid: "a66b3f431af44105ac0df4a076b15063"
format_stamp: "Formatted at 2026-09-28 20:22:00 on dist-test-slave-b83v"
server_key: "d789dfb5977c30c18014e465026ad1d0"
server_key_iv: "ab8b866b3605f61899cbbad1463e04d4"
server_key_version: "kuduclusterkey@0"
I20260928 20:22:00.060712 15728 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20260928 20:22:00.091967 15744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:22:00.092273 15728 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260928 20:22:00.092381 15728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-2/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-2/wal
uuid: "a66b3f431af44105ac0df4a076b15063"
format_stamp: "Formatted at 2026-09-28 20:22:00 on dist-test-slave-b83v"
server_key: "d789dfb5977c30c18014e465026ad1d0"
server_key_iv: "ab8b866b3605f61899cbbad1463e04d4"
server_key_version: "kuduclusterkey@0"
I20260928 20:22:00.092494 15728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:22:00.114928 15728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:22:00.117646 15728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:22:00.117904 15728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:22:00.118244 15728 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:22:00.118707 15728 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:22:00.118808 15728 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:22:00.118880 15728 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:22:00.118940 15728 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:22:00.125721 15728 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.3:33239
I20260928 20:22:00.126305 15728 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
I20260928 20:22:00.126579 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 15728
I20260928 20:22:00.126680 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-2/wal/instance
I20260928 20:22:00.126868 3144 external_mini_cluster.cc:1428] Setting key fda3f59fbd561aebaa3ece4f2840fbfa
I20260928 20:22:00.133250 15857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.3:33239 every 8 connection(s)
Sep 28 20:22:00 dist-test-slave-b83v krb5kdc[14898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626920, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.3@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:22:00.146189 15587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:00.134145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:44995 (local address 127.3.18.62:39369)
0928 20:22:00.134196 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:00.134198 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:00.134552 (+ 354us) server_negotiation.cc:407] Connection header received
0928 20:22:00.135247 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:00.135249 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:00.135263 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:00.135296 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:22:00.136250 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:00.137080 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:00.138631 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:00.138924 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:00.138968 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:00.142094 (+ 3126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:22:00.142097 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:22:00.142097 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:22:00.142107 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:22:00.144211 (+ 2104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:00.144853 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:00.144854 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:00.144855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:00.144925 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:00.145264 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:00.145265 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:00.145265 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:00.145415 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:22:00.145460 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:00.145952 (+ 492us) server_negotiation.cc:299] Negotiation successful
0928 20:22:00.146089 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 20:22:00.146912 15860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:00.134136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:39369 (local address 127.3.18.3:44995)
0928 20:22:00.134372 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:00.134430 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 20:22:00.135103 (+ 673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:22:00.135503 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:22:00.135508 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:22:00.135817 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:22:00.136131 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:00.136138 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:00.137209 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:00.137210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:00.138528 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:00.138532 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:00.139134 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:00.139135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:00.139309 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:00.140056 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:22:00.140067 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:22:00.141940 (+ 1873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:22:00.144332 (+ 2392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:22:00.144344 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:22:00.144348 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:22:00.144727 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:22:00.145012 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:22:00.145013 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:22:00.145013 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:22:00.145180 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:22:00.145513 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:22:00.145520 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:22:00.145735 (+ 215us) client_negotiation.cc:769] Sending connection context
0928 20:22:00.146402 (+ 667us) client_negotiation.cc:240] Negotiation successful
0928 20:22:00.146651 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":132,"threads_started":1}
I20260928 20:22:00.147744 15858 heartbeater.cc:347] Connected to a master server at 127.3.18.62:39369
I20260928 20:22:00.147897 15858 heartbeater.cc:464] Registering TS with master...
I20260928 20:22:00.148139 15858 heartbeater.cc:511] Master 127.3.18.62:39369 requested a full tablet report, sending...
I20260928 20:22:00.148576 15374 ts_manager.cc:194] Registered new tserver with Master: a66b3f431af44105ac0df4a076b15063 (127.3.18.3:33239)
I20260928 20:22:00.149096 15374 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.18.3@KRBTEST.COM'} at 127.3.18.3:44995
I20260928 20:22:00.149991 3144 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 20:22:00.150782 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 15455
I20260928 20:22:00.156957 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 15592
I20260928 20:22:00.162616 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 15728
I20260928 20:22:00.167501 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 15342
I20260928 20:22:00.174373 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:39369
--webserver_interface=127.3.18.62
--webserver_port=34983
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:44943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.18.62:39369
--ranger_config_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.18.20:45363/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.18.20:45363/kms with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:22:00.260571 15861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:22:00.260849 15861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:22:00.260888 15861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:22:00.262151 15861 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:22:00.262235 15861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:22:00.262259 15861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:22:00.262279 15861 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:22:00.262307 15861 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:22:00.264068 15861 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:44943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.18.20:45363/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.18.62:39369
--ranger_config_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:39369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.18.62
--webserver_port=34983
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:22:00.264366 15861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:22:00.264604 15861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:22:00.270115 15866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:22:00.270098 15869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:22:00.270099 15867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:22:00.270401 15861 server_base.cc:1034] running on GCE node
I20260928 20:22:00.270709 15861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:22:00.271054 15861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:22:00.272257 15861 hybrid_clock.cc:648] HybridClock initialized: now 1790626920272225 us; error 52 us; skew 500 ppm
Sep 28 20:22:00 dist-test-slave-b83v krb5kdc[14898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626920, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:22:00.275110 15861 init.cc:377] Logged in from keytab as kudu/127.3.18.62@KRBTEST.COM (short username kudu)
I20260928 20:22:00.275552 15861 webserver.cc:460] Webserver started at http://127.3.18.62:34983/ using document root <none> and password file <none>
I20260928 20:22:00.275763 15861 fs_manager.cc:359] Metadata directory not provided
I20260928 20:22:00.275877 15861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 20:22:00 dist-test-slave-b83v krb5kdc[14898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626920, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.62@KRBTEST.COM for HTTP/127.3.18.20@KRBTEST.COM
I20260928 20:22:00.328490 15861 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 20:22:00.329589 15877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:22:00.329814 15861 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:22:00.329944 15861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/master-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/master-0/wal
uuid: "7f803e5c20bf4ca781ffa4741ee68854"
format_stamp: "Formatted at 2026-09-28 20:21:58 on dist-test-slave-b83v"
server_key: "2c0c7c3c99db083629c90bf86329ca2f"
server_key_iv: "5e39a4781693fb17aaa66f86167ae72f"
server_key_version: "kuduclusterkey@0"
I20260928 20:22:00.330453 15861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:22:00.339231 15861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:22:00.341746 15861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:22:00.341878 15861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:22:00.346370 15861 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.62:39369
I20260928 20:22:00.346432 15929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.62:39369 every 8 connection(s)
I20260928 20:22:00.346860 15861 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
I20260928 20:22:00.348325 15930 sys_catalog.cc:263] Verifying existing consensus state
I20260928 20:22:00.349042 15930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f803e5c20bf4ca781ffa4741ee68854: Bootstrap starting.
I20260928 20:22:00.351266 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 15861
I20260928 20:22:00.351619 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.18.20:45363/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.1:41821
--local_ip_for_outbound_sockets=127.3.18.1
--tserver_master_addrs=127.3.18.62:39369
--webserver_port=40183
--webserver_interface=127.3.18.1
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab
--principal=kudu/127.3.18.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:44943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.18.20:45363/kms with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 20:22:00.358282 15930 log.cc:824] T 00000000000000000000000000000000 P 7f803e5c20bf4ca781ffa4741ee68854: Log is configured to *not* fsync() on all Append() calls
I20260928 20:22:00.360334 15930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f803e5c20bf4ca781ffa4741ee68854: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 20:22:00.360668 15930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f803e5c20bf4ca781ffa4741ee68854: Bootstrap complete.
I20260928 20:22:00.363185 15930 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f803e5c20bf4ca781ffa4741ee68854 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f803e5c20bf4ca781ffa4741ee68854" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 39369 } }
I20260928 20:22:00.363459 15930 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f803e5c20bf4ca781ffa4741ee68854 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f803e5c20bf4ca781ffa4741ee68854, State: Initialized, Role: FOLLOWER
I20260928 20:22:00.363668 15930 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f803e5c20bf4ca781ffa4741ee68854 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f803e5c20bf4ca781ffa4741ee68854" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 39369 } }
I20260928 20:22:00.363803 15930 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f803e5c20bf4ca781ffa4741ee68854 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:22:00.363871 15930 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f803e5c20bf4ca781ffa4741ee68854 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:22:00.363947 15930 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f803e5c20bf4ca781ffa4741ee68854 [term 1 FOLLOWER]: Advancing to term 2
I20260928 20:22:00.365065 15930 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f803e5c20bf4ca781ffa4741ee68854 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f803e5c20bf4ca781ffa4741ee68854" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 39369 } }
I20260928 20:22:00.365183 15930 leader_election.cc:302] T 00000000000000000000000000000000 P 7f803e5c20bf4ca781ffa4741ee68854 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f803e5c20bf4ca781ffa4741ee68854; no voters:
I20260928 20:22:00.365403 15930 leader_election.cc:288] T 00000000000000000000000000000000 P 7f803e5c20bf4ca781ffa4741ee68854 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 20:22:00.365501 15934 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f803e5c20bf4ca781ffa4741ee68854 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 20:22:00.365625 15934 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f803e5c20bf4ca781ffa4741ee68854 [term 2 LEADER]: Becoming Leader. State: Replica: 7f803e5c20bf4ca781ffa4741ee68854, State: Running, Role: LEADER
I20260928 20:22:00.365761 15930 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f803e5c20bf4ca781ffa4741ee68854 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:22:00.365788 15934 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f803e5c20bf4ca781ffa4741ee68854 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f803e5c20bf4ca781ffa4741ee68854" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 39369 } }
I20260928 20:22:00.366161 15936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f803e5c20bf4ca781ffa4741ee68854 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f803e5c20bf4ca781ffa4741ee68854. Latest consensus state: current_term: 2 leader_uuid: "7f803e5c20bf4ca781ffa4741ee68854" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f803e5c20bf4ca781ffa4741ee68854" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 39369 } } }
I20260928 20:22:00.366245 15936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f803e5c20bf4ca781ffa4741ee68854 [sys.catalog]: This master's current role is: LEADER
I20260928 20:22:00.366458 15935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f803e5c20bf4ca781ffa4741ee68854 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7f803e5c20bf4ca781ffa4741ee68854" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f803e5c20bf4ca781ffa4741ee68854" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 39369 } } }
I20260928 20:22:00.366581 15935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f803e5c20bf4ca781ffa4741ee68854 [sys.catalog]: This master's current role is: LEADER
I20260928 20:22:00.366865 15940 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:22:00.367388 15940 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:22:00.367560 15940 catalog_manager.cc:1259] Loaded cluster ID: 6eb8a040d45e4c249513957f91ab8130
I20260928 20:22:00.367630 15940 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:22:00.367738 15930 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 20:22:00.368993 15940 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:22:00.369200 15940 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7f803e5c20bf4ca781ffa4741ee68854: Loaded TSK: 0
I20260928 20:22:00.369454 15940 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 20:22:00.472633 15932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:22:00.472971 15932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:22:00.473014 15932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:22:00.475323 15932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:22:00.475430 15932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:22:00.475488 15932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.1
I20260928 20:22:00.478300 15932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:44943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.18.20:45363/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab
--principal=kudu/127.3.18.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.1:41821
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.18.1
--webserver_port=40183
--tserver_master_addrs=127.3.18.62:39369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.1
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:22:00.478777 15932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:22:00.479099 15932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 20:22:00.486371 15932 server_base.cc:1034] running on GCE node
W20260928 20:22:00.486282 15967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:22:00.486542 15968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:22:00.486739 15970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:22:00.487072 15932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:22:00.487396 15932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:22:00.490710 15932 hybrid_clock.cc:648] HybridClock initialized: now 1790626920490216 us; error 502 us; skew 500 ppm
Sep 28 20:22:00 dist-test-slave-b83v krb5kdc[14898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626920, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:22:00.494454 15932 init.cc:377] Logged in from keytab as kudu/127.3.18.1@KRBTEST.COM (short username kudu)
I20260928 20:22:00.495157 15932 webserver.cc:460] Webserver started at http://127.3.18.1:40183/ using document root <none> and password file <none>
I20260928 20:22:00.495460 15932 fs_manager.cc:359] Metadata directory not provided
I20260928 20:22:00.495590 15932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 20:22:00 dist-test-slave-b83v krb5kdc[14898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626920, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.1@KRBTEST.COM for HTTP/127.3.18.20@KRBTEST.COM
I20260928 20:22:00.523361 15932 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:22:00.524560 15978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:22:00.524791 15932 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 20:22:00.524883 15932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-0/wal
uuid: "114f7db88e624bf5b08061aad81886a0"
format_stamp: "Formatted at 2026-09-28 20:21:59 on dist-test-slave-b83v"
server_key: "f44741b46fa4d51907bfbe2a1e30d594"
server_key_iv: "475d649207abc5f1319f944c8161e673"
server_key_version: "kuduclusterkey@0"
I20260928 20:22:00.525404 15932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:22:00.565223 15932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:22:00.569300 15932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:22:00.569486 15932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:22:00.569790 15932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:22:00.570276 15932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:22:00.570363 15932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:22:00.570425 15932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:22:00.570472 15932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:22:00.578205 15932 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.1:41821
I20260928 20:22:00.578267 16091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.1:41821 every 8 connection(s)
I20260928 20:22:00.578735 15932 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
I20260928 20:22:00.582016 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 15932
I20260928 20:22:00.582314 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.18.20:45363/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.2:34165
--local_ip_for_outbound_sockets=127.3.18.2
--tserver_master_addrs=127.3.18.62:39369
--webserver_port=36077
--webserver_interface=127.3.18.2
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab
--principal=kudu/127.3.18.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:44943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.18.20:45363/kms with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 20:22:00 dist-test-slave-b83v krb5kdc[14898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626920, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.1@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:22:00.592736 16095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:00.579731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:42185 (local address 127.3.18.62:39369)
0928 20:22:00.579997 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:00.580007 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:00.580229 (+ 222us) server_negotiation.cc:407] Connection header received
0928 20:22:00.580952 (+ 723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:00.580958 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:00.581006 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:00.581166 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:22:00.581755 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:00.582649 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:00.584148 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:00.584868 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:00.584952 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:00.587655 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:22:00.587661 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:22:00.587662 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:22:00.587677 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:22:00.590118 (+ 2441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:00.590753 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:00.590757 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:00.590758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:00.590894 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:00.591288 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:00.591289 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:00.591290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:00.591671 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:22:00.591738 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:00.592277 (+ 539us) server_negotiation.cc:299] Negotiation successful
0928 20:22:00.592449 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":113,"threads_started":1}
I20260928 20:22:00.593875 16094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:00.579669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:39369 (local address 127.3.18.1:42185)
0928 20:22:00.580072 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:00.580108 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 20:22:00.580679 (+ 571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:22:00.581207 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:22:00.581212 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:22:00.581425 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:22:00.581634 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:00.581640 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:00.582770 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:00.582773 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:00.584039 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:00.584043 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:00.585109 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:00.585110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:00.585258 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:00.585905 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:22:00.585916 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:22:00.587508 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:22:00.590237 (+ 2729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:22:00.590248 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:22:00.590250 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:22:00.590637 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:22:00.591032 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:22:00.591033 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:22:00.591033 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:22:00.591197 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:22:00.591847 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:22:00.591855 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:22:00.592044 (+ 189us) client_negotiation.cc:769] Sending connection context
0928 20:22:00.593343 (+ 1299us) client_negotiation.cc:240] Negotiation successful
0928 20:22:00.593565 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":115,"threads_started":1}
I20260928 20:22:00.595120 16092 heartbeater.cc:347] Connected to a master server at 127.3.18.62:39369
I20260928 20:22:00.595229 16092 heartbeater.cc:464] Registering TS with master...
I20260928 20:22:00.595470 16092 heartbeater.cc:511] Master 127.3.18.62:39369 requested a full tablet report, sending...
W20260928 20:22:00.596592 16092 heartbeater.cc:644] Failed to heartbeat to 127.3.18.62:39369 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 20:22:00.596668 16092 heartbeater.cc:464] Registering TS with master...
I20260928 20:22:00.596745 16092 heartbeater.cc:511] Master 127.3.18.62:39369 requested a full tablet report, sending...
I20260928 20:22:00.597210 16092 heartbeater.cc:464] Registering TS with master...
I20260928 20:22:00.597280 16092 heartbeater.cc:511] Master 127.3.18.62:39369 requested a full tablet report, sending...
W20260928 20:22:00.597621 16092 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 20:22:00.692348 16096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:22:00.692574 16096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:22:00.692613 16096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:22:00.693979 16096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:22:00.694087 16096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:22:00.694149 16096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.2
I20260928 20:22:00.696156 16096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:44943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.18.20:45363/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab
--principal=kudu/127.3.18.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.2:34165
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.18.2
--webserver_port=36077
--tserver_master_addrs=127.3.18.62:39369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.2
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:22:00.696442 16096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:22:00.696724 16096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:22:00.702894 16102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:22:00.703063 16101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:22:00.703907 16104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:22:00.706046 16096 server_base.cc:1034] running on GCE node
I20260928 20:22:00.706343 16096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:22:00.706805 16096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:22:00.707998 16096 hybrid_clock.cc:648] HybridClock initialized: now 1790626920707974 us; error 42 us; skew 500 ppm
Sep 28 20:22:00 dist-test-slave-b83v krb5kdc[14898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626920, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:22:00.712267 16096 init.cc:377] Logged in from keytab as kudu/127.3.18.2@KRBTEST.COM (short username kudu)
I20260928 20:22:00.712913 16096 webserver.cc:460] Webserver started at http://127.3.18.2:36077/ using document root <none> and password file <none>
I20260928 20:22:00.713229 16096 fs_manager.cc:359] Metadata directory not provided
I20260928 20:22:00.713292 16096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 20:22:00 dist-test-slave-b83v krb5kdc[14898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626920, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.2@KRBTEST.COM for HTTP/127.3.18.20@KRBTEST.COM
I20260928 20:22:00.734347 16096 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.000s
I20260928 20:22:00.735617 16112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:22:00.735908 16096 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:22:00.736024 16096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-1/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-1/wal
uuid: "1c75cca332aa4d3db5010436100997ae"
format_stamp: "Formatted at 2026-09-28 20:21:59 on dist-test-slave-b83v"
server_key: "7938386a3f04dfcd175449d2d910cb31"
server_key_iv: "cc9cde8aa3ec413a7be1d9946ae5bf57"
server_key_version: "kuduclusterkey@0"
I20260928 20:22:00.736611 16096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:22:00.766301 16096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:22:00.768986 16096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:22:00.769187 16096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:22:00.769467 16096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:22:00.769886 16096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:22:00.769941 16096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:22:00.769987 16096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:22:00.770043 16096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:22:00.777958 16096 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.2:34165
I20260928 20:22:00.778388 16096 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
I20260928 20:22:00.779116 16225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.2:34165 every 8 connection(s)
I20260928 20:22:00.780339 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 16096
I20260928 20:22:00.780665 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.18.20:45363/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.3:33239
--local_ip_for_outbound_sockets=127.3.18.3
--tserver_master_addrs=127.3.18.62:39369
--webserver_port=44227
--webserver_interface=127.3.18.3
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab
--principal=kudu/127.3.18.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:44943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.18.20:45363/kms with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 20:22:00 dist-test-slave-b83v krb5kdc[14898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626920, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.2@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:22:00.790647 16095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:00.779650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:51701 (local address 127.3.18.62:39369)
0928 20:22:00.779710 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:00.779712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:00.779723 (+ 11us) server_negotiation.cc:407] Connection header received
0928 20:22:00.780286 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:00.780288 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:00.780306 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:00.780346 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:22:00.781259 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:00.782023 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:00.783155 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:00.783384 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:00.783481 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:00.786513 (+ 3032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:22:00.786515 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:22:00.786516 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:22:00.786526 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:22:00.788625 (+ 2099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:00.789266 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:00.789268 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:00.789268 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:00.789341 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:00.789717 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:00.789718 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:00.789719 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:00.789891 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:22:00.789975 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:00.790400 (+ 425us) server_negotiation.cc:299] Negotiation successful
0928 20:22:00.790530 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 20:22:00.790971 16228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:00.779253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:39369 (local address 127.3.18.2:51701)
0928 20:22:00.779468 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:00.779514 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 20:22:00.780138 (+ 624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:22:00.780560 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:22:00.780565 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:22:00.780836 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:22:00.781130 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:00.781136 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:00.782166 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:00.782167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:00.783038 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:00.783040 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:00.783571 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:00.783572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:00.783727 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:00.784468 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:22:00.784479 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:22:00.786366 (+ 1887us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:22:00.788750 (+ 2384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:22:00.788760 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:22:00.788763 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:22:00.789160 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:22:00.789429 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:22:00.789430 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:22:00.789430 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:22:00.789619 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:22:00.789980 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:22:00.789988 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:22:00.790196 (+ 208us) client_negotiation.cc:769] Sending connection context
0928 20:22:00.790363 (+ 167us) client_negotiation.cc:240] Negotiation successful
0928 20:22:00.790585 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":138,"threads_started":1}
I20260928 20:22:00.791637 16226 heartbeater.cc:347] Connected to a master server at 127.3.18.62:39369
I20260928 20:22:00.791811 16226 heartbeater.cc:464] Registering TS with master...
I20260928 20:22:00.792059 16226 heartbeater.cc:511] Master 127.3.18.62:39369 requested a full tablet report, sending...
W20260928 20:22:00.792937 16226 heartbeater.cc:644] Failed to heartbeat to 127.3.18.62:39369 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 20:22:00.793023 16226 heartbeater.cc:464] Registering TS with master...
I20260928 20:22:00.793104 16226 heartbeater.cc:511] Master 127.3.18.62:39369 requested a full tablet report, sending...
I20260928 20:22:00.793540 16226 heartbeater.cc:464] Registering TS with master...
I20260928 20:22:00.793613 16226 heartbeater.cc:511] Master 127.3.18.62:39369 requested a full tablet report, sending...
W20260928 20:22:00.793926 16226 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 20:22:00.887511 16229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:22:00.887780 16229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:22:00.887835 16229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:22:00.889710 16229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:22:00.889772 16229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:22:00.889815 16229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.3
I20260928 20:22:00.892081 16229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:44943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.18.20:45363/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab
--principal=kudu/127.3.18.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.3:33239
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.18.3
--webserver_port=44227
--tserver_master_addrs=127.3.18.62:39369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.3
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:22:00.892382 16229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:22:00.892666 16229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:22:00.900620 16235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:22:00.900970 16229 server_base.cc:1034] running on GCE node
W20260928 20:22:00.901175 16236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:22:00.901752 16238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:22:00.902144 16229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:22:00.902469 16229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:22:00.903745 16229 hybrid_clock.cc:648] HybridClock initialized: now 1790626920903717 us; error 48 us; skew 500 ppm
Sep 28 20:22:00 dist-test-slave-b83v krb5kdc[14898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626920, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:22:00.906621 16229 init.cc:377] Logged in from keytab as kudu/127.3.18.3@KRBTEST.COM (short username kudu)
I20260928 20:22:00.907291 16229 webserver.cc:460] Webserver started at http://127.3.18.3:44227/ using document root <none> and password file <none>
I20260928 20:22:00.907541 16229 fs_manager.cc:359] Metadata directory not provided
I20260928 20:22:00.907632 16229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 20:22:00 dist-test-slave-b83v krb5kdc[14898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626920, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.3@KRBTEST.COM for HTTP/127.3.18.20@KRBTEST.COM
I20260928 20:22:00.924713 16229 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.001s
I20260928 20:22:00.929738 16246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:22:00.929960 16229 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260928 20:22:00.930068 16229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-2/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-2/wal
uuid: "a66b3f431af44105ac0df4a076b15063"
format_stamp: "Formatted at 2026-09-28 20:22:00 on dist-test-slave-b83v"
server_key: "d789dfb5977c30c18014e465026ad1d0"
server_key_iv: "ab8b866b3605f61899cbbad1463e04d4"
server_key_version: "kuduclusterkey@0"
I20260928 20:22:00.930601 16229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:22:00.963613 16229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:22:00.966559 16229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:22:00.966809 16229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:22:00.967094 16229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:22:00.967548 16229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:22:00.967621 16229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:22:00.967669 16229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:22:00.967703 16229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:22:00.976327 16229 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.3:33239
I20260928 20:22:00.976498 16359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.3:33239 every 8 connection(s)
I20260928 20:22:00.976902 16229 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
I20260928 20:22:00.981953 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 16229
Sep 28 20:22:00 dist-test-slave-b83v krb5kdc[14898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626920, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.3@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:22:00.988891 16095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:00.977983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:55495 (local address 127.3.18.62:39369)
0928 20:22:00.978284 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:00.978287 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:00.978364 (+ 77us) server_negotiation.cc:407] Connection header received
0928 20:22:00.979080 (+ 716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:00.979082 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:00.979095 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:00.979127 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:22:00.980151 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:00.980908 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:00.981970 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:00.982285 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:00.982362 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:00.985324 (+ 2962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:22:00.985326 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:22:00.985327 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:22:00.985336 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:22:00.987074 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:00.987623 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:00.987624 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:00.987625 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:00.987683 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:00.988022 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:00.988023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:00.988024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:00.988189 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:22:00.988235 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:00.988698 (+ 463us) server_negotiation.cc:299] Negotiation successful
0928 20:22:00.988788 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271}
I20260928 20:22:00.989768 16362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:00.977992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:39369 (local address 127.3.18.3:55495)
0928 20:22:00.978208 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:00.978277 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0928 20:22:00.978943 (+ 666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:22:00.979321 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:22:00.979325 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:22:00.979676 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:22:00.979998 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:00.980006 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:00.981020 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:00.981021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:00.981865 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:00.981868 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:00.982408 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:00.982410 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:00.982597 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:00.983257 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:22:00.983268 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:22:00.985200 (+ 1932us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:22:00.987173 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:22:00.987182 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:22:00.987185 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:22:00.987526 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:22:00.987753 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:22:00.987754 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:22:00.987754 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:22:00.987934 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:22:00.988288 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:22:00.988294 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:22:00.988485 (+ 191us) client_negotiation.cc:769] Sending connection context
0928 20:22:00.989303 (+ 818us) client_negotiation.cc:240] Negotiation successful
0928 20:22:00.989501 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":116,"threads_started":1}
I20260928 20:22:00.990504 16360 heartbeater.cc:347] Connected to a master server at 127.3.18.62:39369
I20260928 20:22:00.990617 16360 heartbeater.cc:464] Registering TS with master...
I20260928 20:22:00.990828 16360 heartbeater.cc:511] Master 127.3.18.62:39369 requested a full tablet report, sending...
W20260928 20:22:00.991564 16360 heartbeater.cc:644] Failed to heartbeat to 127.3.18.62:39369 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 20:22:00.991639 16360 heartbeater.cc:464] Registering TS with master...
I20260928 20:22:00.991706 16360 heartbeater.cc:511] Master 127.3.18.62:39369 requested a full tablet report, sending...
I20260928 20:22:00.992143 16360 heartbeater.cc:464] Registering TS with master...
I20260928 20:22:00.992216 16360 heartbeater.cc:511] Master 127.3.18.62:39369 requested a full tablet report, sending...
W20260928 20:22:00.992563 16360 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 20:22:00.997318 16363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:00.982503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52086 (local address 127.3.18.62:39369)
0928 20:22:00.982812 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:00.982813 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:00.982818 (+ 5us) server_negotiation.cc:407] Connection header received
0928 20:22:00.982862 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:00.982863 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:00.982876 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:00.983542 (+ 666us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:22:00.987670 (+ 4128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:00.988470 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:00.989618 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:00.989940 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:00.990004 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:00.991568 (+ 1564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:22:00.991570 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:22:00.991570 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:22:00.991578 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:22:00.993395 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:00.996229 (+ 2834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:00.996231 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:00.996231 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:00.996318 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:00.996617 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:00.996618 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:00.996619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:00.996777 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:22:00.996960 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:00.997090 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 20:22:00.997194 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":72,"threads_started":1}
Sep 28 20:22:01 dist-test-slave-b83v krb5kdc[14898](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790626921, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:22:01.597836 16092 heartbeater.cc:464] Registering TS with master...
I20260928 20:22:01.598017 16092 heartbeater.cc:511] Master 127.3.18.62:39369 requested a full tablet report, sending...
I20260928 20:22:01.598510 15894 ts_manager.cc:194] Registered new tserver with Master: 114f7db88e624bf5b08061aad81886a0 (127.3.18.1:41821)
I20260928 20:22:01.599561 15894 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.18.1@KRBTEST.COM'} at 127.3.18.1:42185
I20260928 20:22:01.794126 16226 heartbeater.cc:464] Registering TS with master...
I20260928 20:22:01.794308 16226 heartbeater.cc:511] Master 127.3.18.62:39369 requested a full tablet report, sending...
I20260928 20:22:01.794713 15894 ts_manager.cc:194] Registered new tserver with Master: 1c75cca332aa4d3db5010436100997ae (127.3.18.2:34165)
I20260928 20:22:01.795286 15894 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.18.2@KRBTEST.COM'} at 127.3.18.2:51701
Sep 28 20:22:01 dist-test-slave-b83v krb5kdc[14898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626921, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.62@KRBTEST.COM for HTTP/127.3.18.20@KRBTEST.COM
I20260928 20:22:01.992735 16360 heartbeater.cc:464] Registering TS with master...
I20260928 20:22:01.992933 16360 heartbeater.cc:511] Master 127.3.18.62:39369 requested a full tablet report, sending...
I20260928 20:22:01.993300 15894 ts_manager.cc:194] Registered new tserver with Master: a66b3f431af44105ac0df4a076b15063 (127.3.18.3:33239)
I20260928 20:22:01.993824 15894 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.18.3@KRBTEST.COM'} at 127.3.18.3:55495
I20260928 20:22:01.998601 3144 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 20:22:02.016575 16381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:01.999996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52098 (local address 127.3.18.62:39369)
0928 20:22:02.003921 (+ 3925us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:02.003924 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:02.003937 (+ 13us) server_negotiation.cc:407] Connection header received
0928 20:22:02.003974 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:02.003975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:02.003992 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:02.004088 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:22:02.006123 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:02.006894 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:02.008972 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:02.009300 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:02.009391 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:02.011269 (+ 1878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:22:02.011271 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:22:02.011272 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:22:02.011282 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:22:02.014363 (+ 3081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:02.014866 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:02.014867 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:02.014868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:02.014945 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:02.015180 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:02.015181 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:02.015181 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:02.015553 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:22:02.015668 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:02.016362 (+ 694us) server_negotiation.cc:299] Negotiation successful
0928 20:22:02.016463 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3890,"thread_start_us":89,"threads_started":1}
I20260928 20:22:02.019543 15894 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52098:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 20:22:02.023834 15894 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 20:22:02.040377 16388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:02.033379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.2:34165 (local address 127.0.0.1:41902)
0928 20:22:02.033996 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:02.034044 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 20:22:02.034118 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:22:02.034885 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:22:02.034889 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:22:02.034891 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:22:02.034980 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:02.034983 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:02.036384 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:02.036385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:02.037730 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:02.037734 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:02.040035 (+ 2301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:02.040036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:02.040124 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:02.040143 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 20:22:02.040217 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 20:22:02.040276 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":88,"threads_started":1}
I20260928 20:22:02.045974 16391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:02.033666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41902 (local address 127.3.18.2:34165)
0928 20:22:02.034431 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:02.034436 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:02.034448 (+ 12us) server_negotiation.cc:407] Connection header received
0928 20:22:02.034489 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:02.034491 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:02.034541 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:02.034738 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:22:02.035083 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:02.036266 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:02.038539 (+ 2273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:02.039367 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:02.039457 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:02.039541 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:02.045783 (+ 6242us) server_negotiation.cc:299] Negotiation successful
0928 20:22:02.045860 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":685,"thread_start_us":66,"threads_started":1}
I20260928 20:22:02.047622 16389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:02.033399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.3:33239 (local address 127.0.0.1:43800)
0928 20:22:02.034226 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:02.034235 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 20:22:02.034272 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:22:02.038184 (+ 3912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:22:02.038185 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:22:02.038188 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:22:02.038234 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:02.038235 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:02.041934 (+ 3699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:02.041936 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:02.043014 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:02.043017 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:02.047348 (+ 4331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:02.047349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:02.047424 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:02.047429 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 20:22:02.047490 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 20:22:02.047536 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":812,"thread_start_us":94,"threads_started":1}
I20260928 20:22:02.048945 16160 tablet_service.cc:1467] Processing CreateTablet for tablet 3ec464bca9ed4a6c9f387516ccca23ea (DEFAULT_TABLE table=test-table [id=02eb83e6df36482893b0e15c2ae6d1f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:22:02.049422 16160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ec464bca9ed4a6c9f387516ccca23ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:22:02.050082 16393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:02.033797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43278 (local address 127.3.18.1:41821)
0928 20:22:02.033919 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:02.033925 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:02.038428 (+ 4503us) server_negotiation.cc:407] Connection header received
0928 20:22:02.039636 (+ 1208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:02.039640 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:02.039660 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:02.039719 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:22:02.043157 (+ 3438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:02.044245 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:02.048495 (+ 4250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:02.049413 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:02.049599 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:02.049670 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:02.049944 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 20:22:02.050000 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":51,"threads_started":1}
I20260928 20:22:02.050537 16392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:02.033675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.1:41821 (local address 127.0.0.1:43278)
0928 20:22:02.038352 (+ 4677us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:02.038364 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:22:02.038461 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:22:02.039899 (+ 1438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:22:02.039900 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:22:02.039902 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:22:02.039959 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:02.039961 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:02.044367 (+ 4406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:02.044368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:02.045477 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:02.045481 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:02.049743 (+ 4262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:02.049744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:02.049831 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:02.049837 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 20:22:02.050403 (+ 566us) client_negotiation.cc:240] Negotiation successful
0928 20:22:02.050458 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4653,"thread_start_us":81,"threads_started":1}
I20260928 20:22:02.051378 16390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:02.033568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43800 (local address 127.3.18.3:33239)
0928 20:22:02.037921 (+ 4353us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:02.037926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:02.037939 (+ 13us) server_negotiation.cc:407] Connection header received
0928 20:22:02.037978 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:02.037980 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:02.037999 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:02.038084 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:22:02.040730 (+ 2646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:02.041799 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:02.046284 (+ 4485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:02.047136 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:02.047225 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:02.047295 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:02.051165 (+ 3870us) server_negotiation.cc:299] Negotiation successful
0928 20:22:02.051293 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4309,"thread_start_us":54,"threads_started":1}
I20260928 20:22:02.052167 16294 tablet_service.cc:1467] Processing CreateTablet for tablet 3ec464bca9ed4a6c9f387516ccca23ea (DEFAULT_TABLE table=test-table [id=02eb83e6df36482893b0e15c2ae6d1f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:22:02.052170 16026 tablet_service.cc:1467] Processing CreateTablet for tablet 3ec464bca9ed4a6c9f387516ccca23ea (DEFAULT_TABLE table=test-table [id=02eb83e6df36482893b0e15c2ae6d1f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:22:02.052557 16294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ec464bca9ed4a6c9f387516ccca23ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:22:02.052557 16026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ec464bca9ed4a6c9f387516ccca23ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:22:02.059995 16395 tablet_bootstrap.cc:492] T 3ec464bca9ed4a6c9f387516ccca23ea P 114f7db88e624bf5b08061aad81886a0: Bootstrap starting.
I20260928 20:22:02.060796 16395 tablet_bootstrap.cc:654] T 3ec464bca9ed4a6c9f387516ccca23ea P 114f7db88e624bf5b08061aad81886a0: Neither blocks nor log segments found. Creating new log.
I20260928 20:22:02.061187 16395 log.cc:824] T 3ec464bca9ed4a6c9f387516ccca23ea P 114f7db88e624bf5b08061aad81886a0: Log is configured to *not* fsync() on all Append() calls
I20260928 20:22:02.061596 16394 tablet_bootstrap.cc:492] T 3ec464bca9ed4a6c9f387516ccca23ea P a66b3f431af44105ac0df4a076b15063: Bootstrap starting.
I20260928 20:22:02.062310 16394 tablet_bootstrap.cc:654] T 3ec464bca9ed4a6c9f387516ccca23ea P a66b3f431af44105ac0df4a076b15063: Neither blocks nor log segments found. Creating new log.
I20260928 20:22:02.062671 16394 log.cc:824] T 3ec464bca9ed4a6c9f387516ccca23ea P a66b3f431af44105ac0df4a076b15063: Log is configured to *not* fsync() on all Append() calls
I20260928 20:22:02.063051 16396 tablet_bootstrap.cc:492] T 3ec464bca9ed4a6c9f387516ccca23ea P 1c75cca332aa4d3db5010436100997ae: Bootstrap starting.
I20260928 20:22:02.063730 16396 tablet_bootstrap.cc:654] T 3ec464bca9ed4a6c9f387516ccca23ea P 1c75cca332aa4d3db5010436100997ae: Neither blocks nor log segments found. Creating new log.
I20260928 20:22:02.064113 16396 log.cc:824] T 3ec464bca9ed4a6c9f387516ccca23ea P 1c75cca332aa4d3db5010436100997ae: Log is configured to *not* fsync() on all Append() calls
I20260928 20:22:02.068701 16394 tablet_bootstrap.cc:492] T 3ec464bca9ed4a6c9f387516ccca23ea P a66b3f431af44105ac0df4a076b15063: No bootstrap required, opened a new log
I20260928 20:22:02.068822 16394 ts_tablet_manager.cc:1397] T 3ec464bca9ed4a6c9f387516ccca23ea P a66b3f431af44105ac0df4a076b15063: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260928 20:22:02.071636 16394 raft_consensus.cc:348] T 3ec464bca9ed4a6c9f387516ccca23ea P a66b3f431af44105ac0df4a076b15063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c75cca332aa4d3db5010436100997ae" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34165 } } peers { permanent_uuid: "114f7db88e624bf5b08061aad81886a0" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 41821 } } peers { permanent_uuid: "a66b3f431af44105ac0df4a076b15063" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 33239 } }
I20260928 20:22:02.071923 16394 raft_consensus.cc:374] T 3ec464bca9ed4a6c9f387516ccca23ea P a66b3f431af44105ac0df4a076b15063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:22:02.071985 16394 raft_consensus.cc:729] T 3ec464bca9ed4a6c9f387516ccca23ea P a66b3f431af44105ac0df4a076b15063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a66b3f431af44105ac0df4a076b15063, State: Initialized, Role: FOLLOWER
I20260928 20:22:02.072688 16395 tablet_bootstrap.cc:492] T 3ec464bca9ed4a6c9f387516ccca23ea P 114f7db88e624bf5b08061aad81886a0: No bootstrap required, opened a new log
I20260928 20:22:02.072798 16395 ts_tablet_manager.cc:1397] T 3ec464bca9ed4a6c9f387516ccca23ea P 114f7db88e624bf5b08061aad81886a0: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20260928 20:22:02.075661 16395 raft_consensus.cc:348] T 3ec464bca9ed4a6c9f387516ccca23ea P 114f7db88e624bf5b08061aad81886a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c75cca332aa4d3db5010436100997ae" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34165 } } peers { permanent_uuid: "114f7db88e624bf5b08061aad81886a0" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 41821 } } peers { permanent_uuid: "a66b3f431af44105ac0df4a076b15063" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 33239 } }
I20260928 20:22:02.076442 16395 raft_consensus.cc:374] T 3ec464bca9ed4a6c9f387516ccca23ea P 114f7db88e624bf5b08061aad81886a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:22:02.076514 16395 raft_consensus.cc:729] T 3ec464bca9ed4a6c9f387516ccca23ea P 114f7db88e624bf5b08061aad81886a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 114f7db88e624bf5b08061aad81886a0, State: Initialized, Role: FOLLOWER
I20260928 20:22:02.076695 16395 consensus_queue.cc:260] T 3ec464bca9ed4a6c9f387516ccca23ea P 114f7db88e624bf5b08061aad81886a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c75cca332aa4d3db5010436100997ae" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34165 } } peers { permanent_uuid: "114f7db88e624bf5b08061aad81886a0" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 41821 } } peers { permanent_uuid: "a66b3f431af44105ac0df4a076b15063" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 33239 } }
I20260928 20:22:02.077064 16395 ts_tablet_manager.cc:1428] T 3ec464bca9ed4a6c9f387516ccca23ea P 114f7db88e624bf5b08061aad81886a0: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260928 20:22:02.077217 16092 heartbeater.cc:503] Master 127.3.18.62:39369 was elected leader, sending a full tablet report...
I20260928 20:22:02.078409 16394 consensus_queue.cc:260] T 3ec464bca9ed4a6c9f387516ccca23ea P a66b3f431af44105ac0df4a076b15063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c75cca332aa4d3db5010436100997ae" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34165 } } peers { permanent_uuid: "114f7db88e624bf5b08061aad81886a0" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 41821 } } peers { permanent_uuid: "a66b3f431af44105ac0df4a076b15063" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 33239 } }
I20260928 20:22:02.078828 16394 ts_tablet_manager.cc:1428] T 3ec464bca9ed4a6c9f387516ccca23ea P a66b3f431af44105ac0df4a076b15063: Time spent starting tablet: real 0.010s user 0.004s sys 0.000s
I20260928 20:22:02.078985 16360 heartbeater.cc:503] Master 127.3.18.62:39369 was elected leader, sending a full tablet report...
W20260928 20:22:02.080483 16093 tablet.cc:2369] T 3ec464bca9ed4a6c9f387516ccca23ea P 114f7db88e624bf5b08061aad81886a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 20:22:02.082166 16396 tablet_bootstrap.cc:492] T 3ec464bca9ed4a6c9f387516ccca23ea P 1c75cca332aa4d3db5010436100997ae: No bootstrap required, opened a new log
I20260928 20:22:02.082266 16396 ts_tablet_manager.cc:1397] T 3ec464bca9ed4a6c9f387516ccca23ea P 1c75cca332aa4d3db5010436100997ae: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20260928 20:22:02.084666 16396 raft_consensus.cc:348] T 3ec464bca9ed4a6c9f387516ccca23ea P 1c75cca332aa4d3db5010436100997ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c75cca332aa4d3db5010436100997ae" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34165 } } peers { permanent_uuid: "114f7db88e624bf5b08061aad81886a0" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 41821 } } peers { permanent_uuid: "a66b3f431af44105ac0df4a076b15063" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 33239 } }
I20260928 20:22:02.084890 16396 raft_consensus.cc:374] T 3ec464bca9ed4a6c9f387516ccca23ea P 1c75cca332aa4d3db5010436100997ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:22:02.084936 16396 raft_consensus.cc:729] T 3ec464bca9ed4a6c9f387516ccca23ea P 1c75cca332aa4d3db5010436100997ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c75cca332aa4d3db5010436100997ae, State: Initialized, Role: FOLLOWER
I20260928 20:22:02.085106 16396 consensus_queue.cc:260] T 3ec464bca9ed4a6c9f387516ccca23ea P 1c75cca332aa4d3db5010436100997ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c75cca332aa4d3db5010436100997ae" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34165 } } peers { permanent_uuid: "114f7db88e624bf5b08061aad81886a0" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 41821 } } peers { permanent_uuid: "a66b3f431af44105ac0df4a076b15063" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 33239 } }
I20260928 20:22:02.085421 16396 ts_tablet_manager.cc:1428] T 3ec464bca9ed4a6c9f387516ccca23ea P 1c75cca332aa4d3db5010436100997ae: Time spent starting tablet: real 0.003s user 0.001s sys 0.003s
I20260928 20:22:02.085564 16226 heartbeater.cc:503] Master 127.3.18.62:39369 was elected leader, sending a full tablet report...
W20260928 20:22:02.228693 16361 tablet.cc:2369] T 3ec464bca9ed4a6c9f387516ccca23ea P a66b3f431af44105ac0df4a076b15063: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 20:22:02 dist-test-slave-b83v krb5kdc[14898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626921, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.62@KRBTEST.COM for HTTP/127.3.18.20@KRBTEST.COM
W20260928 20:22:02.280670 16227 tablet.cc:2369] T 3ec464bca9ed4a6c9f387516ccca23ea P 1c75cca332aa4d3db5010436100997ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 20:22:02.301077 16402 raft_consensus.cc:482] T 3ec464bca9ed4a6c9f387516ccca23ea P 1c75cca332aa4d3db5010436100997ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:22:02.301244 16402 raft_consensus.cc:504] T 3ec464bca9ed4a6c9f387516ccca23ea P 1c75cca332aa4d3db5010436100997ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c75cca332aa4d3db5010436100997ae" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34165 } } peers { permanent_uuid: "114f7db88e624bf5b08061aad81886a0" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 41821 } } peers { permanent_uuid: "a66b3f431af44105ac0df4a076b15063" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 33239 } }
I20260928 20:22:02.301636 16402 leader_election.cc:288] T 3ec464bca9ed4a6c9f387516ccca23ea P 1c75cca332aa4d3db5010436100997ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 114f7db88e624bf5b08061aad81886a0 (127.3.18.1:41821), a66b3f431af44105ac0df4a076b15063 (127.3.18.3:33239)
I20260928 20:22:02.307674 16390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:02.302135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:57709 (local address 127.3.18.3:33239)
0928 20:22:02.302176 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:02.302177 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:02.302882 (+ 705us) server_negotiation.cc:407] Connection header received
0928 20:22:02.302992 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:02.302993 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:02.303006 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:02.303034 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:22:02.303220 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:02.303845 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:02.306261 (+ 2416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:02.306838 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:02.306883 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:02.306895 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:02.307534 (+ 639us) server_negotiation.cc:299] Negotiation successful
0928 20:22:02.307589 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 20:22:02.308498 16405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:02.301854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.3:33239 (local address 127.3.18.2:57709)
0928 20:22:02.302821 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:02.302833 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:22:02.302909 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:22:02.303093 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:22:02.303093 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:22:02.303095 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:22:02.303157 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:02.303158 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:02.304100 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:02.304100 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:02.306143 (+ 2043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:02.306148 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:02.307356 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:02.307357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:02.307439 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:02.307443 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:22:02.308128 (+ 685us) client_negotiation.cc:240] Negotiation successful
0928 20:22:02.308183 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":949,"thread_start_us":73,"threads_started":1}
I20260928 20:22:02.308918 16393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:02.302036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:57771 (local address 127.3.18.1:41821)
0928 20:22:02.302082 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:02.302085 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:02.302243 (+ 158us) server_negotiation.cc:407] Connection header received
0928 20:22:02.302349 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:02.302351 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:02.302363 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:02.302391 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:22:02.302659 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:02.303370 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:02.306816 (+ 3446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:02.307380 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:02.307426 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:02.307437 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:02.308780 (+ 1343us) server_negotiation.cc:299] Negotiation successful
0928 20:22:02.308837 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 20:22:02.309297 16314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ec464bca9ed4a6c9f387516ccca23ea" candidate_uuid: "1c75cca332aa4d3db5010436100997ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a66b3f431af44105ac0df4a076b15063" is_pre_election: true
I20260928 20:22:02.309518 16314 raft_consensus.cc:2454] T 3ec464bca9ed4a6c9f387516ccca23ea P a66b3f431af44105ac0df4a076b15063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c75cca332aa4d3db5010436100997ae in term 0.
I20260928 20:22:02.309744 16404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:02.301777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.1:41821 (local address 127.3.18.2:57771)
0928 20:22:02.302143 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:02.302194 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 20:22:02.302272 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:22:02.302497 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:22:02.302498 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:22:02.302500 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:22:02.302582 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:02.302584 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:02.303741 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:02.303741 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:02.306707 (+ 2966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:02.306711 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:02.308273 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:02.308274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:02.308349 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:02.308353 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:22:02.309326 (+ 973us) client_negotiation.cc:240] Negotiation successful
0928 20:22:02.309375 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":125,"threads_started":1}
I20260928 20:22:02.309864 16115 leader_election.cc:302] T 3ec464bca9ed4a6c9f387516ccca23ea P 1c75cca332aa4d3db5010436100997ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c75cca332aa4d3db5010436100997ae, a66b3f431af44105ac0df4a076b15063; no voters:
I20260928 20:22:02.310094 16046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ec464bca9ed4a6c9f387516ccca23ea" candidate_uuid: "1c75cca332aa4d3db5010436100997ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "114f7db88e624bf5b08061aad81886a0" is_pre_election: true
I20260928 20:22:02.310216 16046 raft_consensus.cc:2454] T 3ec464bca9ed4a6c9f387516ccca23ea P 114f7db88e624bf5b08061aad81886a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c75cca332aa4d3db5010436100997ae in term 0.
I20260928 20:22:02.310341 16402 raft_consensus.cc:2781] T 3ec464bca9ed4a6c9f387516ccca23ea P 1c75cca332aa4d3db5010436100997ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 20:22:02.310410 16402 raft_consensus.cc:482] T 3ec464bca9ed4a6c9f387516ccca23ea P 1c75cca332aa4d3db5010436100997ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 20:22:02.310433 16402 raft_consensus.cc:3037] T 3ec464bca9ed4a6c9f387516ccca23ea P 1c75cca332aa4d3db5010436100997ae [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:22:02.311141 16402 raft_consensus.cc:504] T 3ec464bca9ed4a6c9f387516ccca23ea P 1c75cca332aa4d3db5010436100997ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c75cca332aa4d3db5010436100997ae" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34165 } } peers { permanent_uuid: "114f7db88e624bf5b08061aad81886a0" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 41821 } } peers { permanent_uuid: "a66b3f431af44105ac0df4a076b15063" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 33239 } }
I20260928 20:22:02.311302 16402 leader_election.cc:288] T 3ec464bca9ed4a6c9f387516ccca23ea P 1c75cca332aa4d3db5010436100997ae [CANDIDATE]: Term 1 election: Requested vote from peers 114f7db88e624bf5b08061aad81886a0 (127.3.18.1:41821), a66b3f431af44105ac0df4a076b15063 (127.3.18.3:33239)
I20260928 20:22:02.311620 16046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ec464bca9ed4a6c9f387516ccca23ea" candidate_uuid: "1c75cca332aa4d3db5010436100997ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "114f7db88e624bf5b08061aad81886a0"
I20260928 20:22:02.311725 16046 raft_consensus.cc:3037] T 3ec464bca9ed4a6c9f387516ccca23ea P 114f7db88e624bf5b08061aad81886a0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:22:02.312083 16314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ec464bca9ed4a6c9f387516ccca23ea" candidate_uuid: "1c75cca332aa4d3db5010436100997ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a66b3f431af44105ac0df4a076b15063"
I20260928 20:22:02.312197 16314 raft_consensus.cc:3037] T 3ec464bca9ed4a6c9f387516ccca23ea P a66b3f431af44105ac0df4a076b15063 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:22:02.312760 16046 raft_consensus.cc:2454] T 3ec464bca9ed4a6c9f387516ccca23ea P 114f7db88e624bf5b08061aad81886a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c75cca332aa4d3db5010436100997ae in term 1.
I20260928 20:22:02.312947 16116 leader_election.cc:302] T 3ec464bca9ed4a6c9f387516ccca23ea P 1c75cca332aa4d3db5010436100997ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 114f7db88e624bf5b08061aad81886a0, 1c75cca332aa4d3db5010436100997ae; no voters:
I20260928 20:22:02.313045 16402 raft_consensus.cc:2781] T 3ec464bca9ed4a6c9f387516ccca23ea P 1c75cca332aa4d3db5010436100997ae [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:22:02.313099 16314 raft_consensus.cc:2454] T 3ec464bca9ed4a6c9f387516ccca23ea P a66b3f431af44105ac0df4a076b15063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c75cca332aa4d3db5010436100997ae in term 1.
I20260928 20:22:02.313213 16402 raft_consensus.cc:686] T 3ec464bca9ed4a6c9f387516ccca23ea P 1c75cca332aa4d3db5010436100997ae [term 1 LEADER]: Becoming Leader. State: Replica: 1c75cca332aa4d3db5010436100997ae, State: Running, Role: LEADER
I20260928 20:22:02.313321 16402 consensus_queue.cc:237] T 3ec464bca9ed4a6c9f387516ccca23ea P 1c75cca332aa4d3db5010436100997ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c75cca332aa4d3db5010436100997ae" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34165 } } peers { permanent_uuid: "114f7db88e624bf5b08061aad81886a0" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 41821 } } peers { permanent_uuid: "a66b3f431af44105ac0df4a076b15063" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 33239 } }
I20260928 20:22:02.314170 15894 catalog_manager.cc:5495] T 3ec464bca9ed4a6c9f387516ccca23ea P 1c75cca332aa4d3db5010436100997ae reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c75cca332aa4d3db5010436100997ae (127.3.18.2). New cstate: current_term: 1 leader_uuid: "1c75cca332aa4d3db5010436100997ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c75cca332aa4d3db5010436100997ae" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 34165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "114f7db88e624bf5b08061aad81886a0" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 41821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a66b3f431af44105ac0df4a076b15063" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 33239 } health_report { overall_health: UNKNOWN } } }
I20260928 20:22:02.400142 16391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:02.396431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42672 (local address 127.3.18.2:34165)
0928 20:22:02.396488 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:02.396491 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:02.396506 (+ 15us) server_negotiation.cc:407] Connection header received
0928 20:22:02.396585 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:02.396586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:02.396615 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:02.396725 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:22:02.396943 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:02.397741 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:02.398869 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:02.399144 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:02.399213 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:02.399465 (+ 252us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:22:02.399612 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:22:02.399695 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:02.400026 (+ 331us) server_negotiation.cc:299] Negotiation successful
0928 20:22:02.400051 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 20:22:02.409070 16314 raft_consensus.cc:1261] T 3ec464bca9ed4a6c9f387516ccca23ea P a66b3f431af44105ac0df4a076b15063 [term 1 FOLLOWER]: Refusing update from remote peer 1c75cca332aa4d3db5010436100997ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:22:02.409428 16046 raft_consensus.cc:1261] T 3ec464bca9ed4a6c9f387516ccca23ea P 114f7db88e624bf5b08061aad81886a0 [term 1 FOLLOWER]: Refusing update from remote peer 1c75cca332aa4d3db5010436100997ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:22:02.409760 16402 consensus_queue.cc:1035] T 3ec464bca9ed4a6c9f387516ccca23ea P 1c75cca332aa4d3db5010436100997ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "114f7db88e624bf5b08061aad81886a0" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 41821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:22:02.410135 16402 consensus_queue.cc:1035] T 3ec464bca9ed4a6c9f387516ccca23ea P 1c75cca332aa4d3db5010436100997ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "a66b3f431af44105ac0df4a076b15063" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 33239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:22:02.416158 16410 mvcc.cc:205] Tried to move back new op lower bound from 7334407874176004096 to 7334407873795997696. Current Snapshot: MvccSnapshot[applied={T|T < 7334407874176004096}]
I20260928 20:22:02.418097 16411 mvcc.cc:205] Tried to move back new op lower bound from 7334407874176004096 to 7334407873795997696. Current Snapshot: MvccSnapshot[applied={T|T < 7334407874176004096}]
I20260928 20:22:02.418635 16413 mvcc.cc:205] Tried to move back new op lower bound from 7334407874176004096 to 7334407873795997696. Current Snapshot: MvccSnapshot[applied={T|T < 7334407874176004096}]
I20260928 20:22:02.421119 15894 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52098:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:22:02.421227 15894 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52098:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:22:02.421928 15894 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7f803e5c20bf4ca781ffa4741ee68854: Sending DeleteTablet for 3 replicas of tablet 3ec464bca9ed4a6c9f387516ccca23ea
I20260928 20:22:02.422425 16026 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ec464bca9ed4a6c9f387516ccca23ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:22:02 UTC) from {username='kudu', principal='kudu/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:43278
I20260928 20:22:02.422688 16294 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ec464bca9ed4a6c9f387516ccca23ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:22:02 UTC) from {username='kudu', principal='kudu/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:43800
I20260928 20:22:02.423110 16160 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ec464bca9ed4a6c9f387516ccca23ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:22:02 UTC) from {username='kudu', principal='kudu/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:41902
I20260928 20:22:02.423390 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 15932
I20260928 20:22:02.425179 16420 tablet_replica.cc:330] T 3ec464bca9ed4a6c9f387516ccca23ea P a66b3f431af44105ac0df4a076b15063: stopping tablet replica
I20260928 20:22:02.425335 16420 raft_consensus.cc:2229] T 3ec464bca9ed4a6c9f387516ccca23ea P a66b3f431af44105ac0df4a076b15063 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:22:02.425436 16420 raft_consensus.cc:2258] T 3ec464bca9ed4a6c9f387516ccca23ea P a66b3f431af44105ac0df4a076b15063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:22:02.434090 16420 ts_tablet_manager.cc:1905] T 3ec464bca9ed4a6c9f387516ccca23ea P a66b3f431af44105ac0df4a076b15063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:22:02.436519 16420 ts_tablet_manager.cc:1918] T 3ec464bca9ed4a6c9f387516ccca23ea P a66b3f431af44105ac0df4a076b15063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:22:02.436627 16420 log.cc:1196] T 3ec464bca9ed4a6c9f387516ccca23ea P a66b3f431af44105ac0df4a076b15063: Deleting WAL directory at /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790626764250411-3144-0/minicluster-data/ts-2/wal/wals/3ec464bca9ed4a6c9f387516ccca23ea
I20260928 20:22:02.436935 16420 ts_tablet_manager.cc:1939] T 3ec464bca9ed4a6c9f387516ccca23ea P a66b3f431af44105ac0df4a076b15063: Deleting consensus metadata
I20260928 20:22:02.437454 15880 catalog_manager.cc:4841] TS a66b3f431af44105ac0df4a076b15063 (127.3.18.3:33239): tablet 3ec464bca9ed4a6c9f387516ccca23ea (table test-table [id=02eb83e6df36482893b0e15c2ae6d1f0]) successfully deleted
W20260928 20:22:02.437731 15881 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.18.1:41821 (error 108)
I20260928 20:22:02.440735 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 16096
I20260928 20:22:02.444403 16392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:02.444255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.1:41821 (local address 127.0.0.1:44058)
0928 20:22:02.444313 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:02.444359 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.1:41821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 20:22:02.444607 15881 catalog_manager.cc:4584] TS 114f7db88e624bf5b08061aad81886a0 (127.3.18.1:41821): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3ec464bca9ed4a6c9f387516ccca23ea: Network error: Client connection negotiation failed: client connection to 127.3.18.1:41821: connect: Connection refused (error 111)
I20260928 20:22:02.449559 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 16229
I20260928 20:22:02.455425 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 15861
I20260928 20:22:02.464380 3144 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 20:22:02.810065 3144 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 20:22:02.810207 3144 mini_ranger.cc:66] Stopping Ranger...
I20260928 20:22:03.155606 3144 mini_ranger.cc:68] Stopped Ranger
2026-09-28 20:22:03.155 UTC [14928] LOG: received smart shutdown request
2026-09-28 20:22:03.160 UTC [14928] LOG: background worker "logical replication launcher" (PID 14937) exited with exit code 1
2026-09-28 20:22:03.160 UTC [14932] LOG: shutting down
2026-09-28 20:22:03.394 UTC [14928] LOG: database system is shut down
2026-09-28T20:22:03Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (51916 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:22:03 dist-test-slave-b83v krb5kdc[16431](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:22:03 dist-test-slave-b83v krb5kdc[16431](info): set up 2 sockets
Sep 28 20:22:03 dist-test-slave-b83v krb5kdc[16431](info): commencing operation
krb5kdc: starting...
W20260928 20:22:04.579995 3144 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Sep 28 20:22:04 dist-test-slave-b83v krb5kdc[16431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:22:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:22:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.18.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab.
Entry for principal kudu/127.3.18.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab.
I20260928 20:22:04.702888 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:36979
--webserver_interface=127.3.18.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:37461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.18.62:36979,127.3.18.61:36447,127.3.18.60:44445
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:22:04.791034 16444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:22:04.791244 16444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:22:04.791266 16444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:22:04.792479 16444 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:22:04.792526 16444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:22:04.792539 16444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:22:04.792551 16444 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:22:04.792670 16444 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:22:04.794255 16444 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:37461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.18.62:36979,127.3.18.61:36447,127.3.18.60:44445
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:36979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.18.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:22:04.794525 16444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:22:04.794804 16444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:22:04.800067 16449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:22:04.800053 16452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:22:04.800053 16450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:22:04.800303 16444 server_base.cc:1034] running on GCE node
I20260928 20:22:04.800554 16444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:22:04.800841 16444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:22:04.801985 16444 hybrid_clock.cc:648] HybridClock initialized: now 1790626924801961 us; error 36 us; skew 500 ppm
Sep 28 20:22:04 dist-test-slave-b83v krb5kdc[16431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626924, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:22:04.804698 16444 init.cc:377] Logged in from keytab as kudu/127.3.18.62@KRBTEST.COM (short username kudu)
I20260928 20:22:04.805197 16444 webserver.cc:460] Webserver started at http://127.3.18.62:34249/ using document root <none> and password file <none>
I20260928 20:22:04.805478 16444 fs_manager.cc:359] Metadata directory not provided
I20260928 20:22:04.805527 16444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:22:04.805667 16444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:22:04.806754 16444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/minicluster-data/master-0/data/instance:
uuid: "0d73bade8df3436da2af29916e688cd0"
format_stamp: "Formatted at 2026-09-28 20:22:04 on dist-test-slave-b83v"
server_key: "2ba26036526c4ffed7a3c649271516d3"
server_key_iv: "b6b8ce2bdf33068289e727b7e3c40f19"
server_key_version: "encryptionkey@0"
I20260928 20:22:04.807117 16444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/minicluster-data/master-0/wal/instance:
uuid: "0d73bade8df3436da2af29916e688cd0"
format_stamp: "Formatted at 2026-09-28 20:22:04 on dist-test-slave-b83v"
server_key: "2ba26036526c4ffed7a3c649271516d3"
server_key_iv: "b6b8ce2bdf33068289e727b7e3c40f19"
server_key_version: "encryptionkey@0"
I20260928 20:22:04.808645 16444 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260928 20:22:04.809535 16459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:22:04.809777 16444 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:22:04.809845 16444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/minicluster-data/master-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/minicluster-data/master-0/wal
uuid: "0d73bade8df3436da2af29916e688cd0"
format_stamp: "Formatted at 2026-09-28 20:22:04 on dist-test-slave-b83v"
server_key: "2ba26036526c4ffed7a3c649271516d3"
server_key_iv: "b6b8ce2bdf33068289e727b7e3c40f19"
server_key_version: "encryptionkey@0"
I20260928 20:22:04.809939 16444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:22:04.837512 16444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:22:04.840210 16444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:22:04.840375 16444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:22:04.845086 16444 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.62:36979
I20260928 20:22:04.845147 16511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.62:36979 every 8 connection(s)
I20260928 20:22:04.845580 16444 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
I20260928 20:22:04.846588 16512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:22:04.848456 16512 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36979 } has no permanent_uuid. Determining permanent_uuid...
I20260928 20:22:04.849133 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 16444
I20260928 20:22:04.849278 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/minicluster-data/master-0/wal/instance
I20260928 20:22:04.849550 3144 external_mini_cluster.cc:1428] Setting key 01884a1c784665d4fd89ec630d3f3cf9
Sep 28 20:22:04 dist-test-slave-b83v krb5kdc[16431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626924, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.62@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.18.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.61@KRBTEST.COM" created.
I20260928 20:22:04.861132 16515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:04.849262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36979 (local address 127.0.0.1:52114)
0928 20:22:04.849479 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:04.849586 (+ 107us) client_negotiation.cc:174] Beginning negotiation
0928 20:22:04.850374 (+ 788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:22:04.850729 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:22:04.850733 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:22:04.851164 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:22:04.851443 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:04.851450 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:04.852615 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:04.852616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:04.853413 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:04.853415 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:04.853946 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:04.853947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:04.854013 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:04.854622 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:22:04.854631 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:22:04.856477 (+ 1846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:22:04.858931 (+ 2454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:22:04.858940 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:22:04.858943 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:22:04.859232 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:22:04.859502 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:22:04.859503 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:22:04.859503 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:22:04.859694 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:22:04.860273 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:22:04.860280 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:22:04.860355 (+ 75us) client_negotiation.cc:769] Sending connection context
0928 20:22:04.860555 (+ 200us) client_negotiation.cc:240] Negotiation successful
0928 20:22:04.860874 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":153,"thread_start_us":97,"threads_started":1}
I20260928 20:22:04.861140 16514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:04.849262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52114 (local address 127.3.18.62:36979)
0928 20:22:04.849480 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:04.849491 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:04.849754 (+ 263us) server_negotiation.cc:407] Connection header received
0928 20:22:04.850570 (+ 816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:04.850572 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:04.850592 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:04.850686 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:22:04.851576 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:04.852498 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:04.853510 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:04.853847 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:04.853970 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:04.856610 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:22:04.856622 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:22:04.856623 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:22:04.856633 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:22:04.858815 (+ 2182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:04.859329 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:04.859333 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:04.859333 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:04.859426 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:04.859847 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:04.859849 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:04.859850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:04.860164 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:22:04.860252 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:04.860614 (+ 362us) server_negotiation.cc:299] Negotiation successful
0928 20:22:04.860817 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":171,"thread_start_us":98,"threads_started":1}
I20260928 20:22:04.862069 16512 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.18.61" port: 36447 } has no permanent_uuid. Determining permanent_uuid...
I20260928 20:22:04.862493 16515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:04.862343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.61:36447 (local address 127.0.0.1:51868)
0928 20:22:04.862395 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:04.862456 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.61:36447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 20:22:04.862627 16462 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.18.61:36447: connect: Connection refused (error 111)
I20260928 20:22:04.863092 16515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:04.862993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.61:36447 (local address 127.0.0.1:51878)
0928 20:22:04.863043 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:04.863069 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.61:36447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 20:22:04.863201 16512 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.18.61:36447: Network error: Client connection negotiation failed: client connection to 127.3.18.61:36447: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.61.keytab.
Entry for principal kudu/127.3.18.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.61.keytab.
I20260928 20:22:04.879407 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.61:36447
--webserver_interface=127.3.18.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.61.keytab
--principal=kudu/127.3.18.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:37461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.18.62:36979,127.3.18.61:36447,127.3.18.60:44445
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 20:22:04.912495 16512 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.18.61" port: 36447 } attempt: 1
I20260928 20:22:04.912992 16515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:04.912854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.61:36447 (local address 127.0.0.1:51894)
0928 20:22:04.912923 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:04.912963 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.61:36447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 20:22:04.913386 16515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:04.913261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.61:36447 (local address 127.0.0.1:51902)
0928 20:22:04.913309 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:04.913342 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.61:36447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 20:22:04.913473 16512 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.18.61:36447: Network error: Client connection negotiation failed: client connection to 127.3.18.61:36447: connect: Connection refused (error 111)
W20260928 20:22:04.976776 16519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:22:04.977047 16519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:22:04.977082 16519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:22:04.978533 16519 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:22:04.978600 16519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:22:04.978619 16519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:22:04.978636 16519 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:22:04.978652 16519 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:22:04.980628 16519 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:37461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.18.62:36979,127.3.18.61:36447,127.3.18.60:44445
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.61.keytab
--principal=kudu/127.3.18.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.61:36447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.18.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:22:04.980981 16519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:22:04.981259 16519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 20:22:04.981649 16512 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.18.61" port: 36447 } attempt: 2
I20260928 20:22:04.982142 16515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:04.981995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.61:36447 (local address 127.0.0.1:51908)
0928 20:22:04.982068 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:04.982111 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.61:36447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 20:22:04.982708 16515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:04.982578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.61:36447 (local address 127.0.0.1:51916)
0928 20:22:04.982649 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:04.982683 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.61:36447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 20:22:04.982847 16512 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.18.61:36447: Network error: Client connection negotiation failed: client connection to 127.3.18.61:36447: connect: Connection refused (error 111)
W20260928 20:22:04.987591 16524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:22:04.987591 16525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:22:04.987713 16519 server_base.cc:1034] running on GCE node
W20260928 20:22:04.987665 16527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:22:04.988169 16519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:22:04.988530 16519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:22:04.989707 16519 hybrid_clock.cc:648] HybridClock initialized: now 1790626924989694 us; error 29 us; skew 500 ppm
Sep 28 20:22:04 dist-test-slave-b83v krb5kdc[16431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626924, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:22:04.992461 16519 init.cc:377] Logged in from keytab as kudu/127.3.18.61@KRBTEST.COM (short username kudu)
I20260928 20:22:04.992990 16519 webserver.cc:460] Webserver started at http://127.3.18.61:37531/ using document root <none> and password file <none>
I20260928 20:22:04.993233 16519 fs_manager.cc:359] Metadata directory not provided
I20260928 20:22:04.993306 16519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:22:04.993462 16519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:22:04.994565 16519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/minicluster-data/master-1/data/instance:
uuid: "5e39f28d7fb349c795d564741dc504fb"
format_stamp: "Formatted at 2026-09-28 20:22:04 on dist-test-slave-b83v"
server_key: "0ebf84bdb29ac329e7b317c4a759a02d"
server_key_iv: "678f9b7d722f94ffe871524b3f810ea5"
server_key_version: "encryptionkey@0"
I20260928 20:22:04.994969 16519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/minicluster-data/master-1/wal/instance:
uuid: "5e39f28d7fb349c795d564741dc504fb"
format_stamp: "Formatted at 2026-09-28 20:22:04 on dist-test-slave-b83v"
server_key: "0ebf84bdb29ac329e7b317c4a759a02d"
server_key_iv: "678f9b7d722f94ffe871524b3f810ea5"
server_key_version: "encryptionkey@0"
I20260928 20:22:04.996483 16519 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:22:04.997342 16534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:22:04.997535 16519 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:22:04.997651 16519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/minicluster-data/master-1/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/minicluster-data/master-1/wal
uuid: "5e39f28d7fb349c795d564741dc504fb"
format_stamp: "Formatted at 2026-09-28 20:22:04 on dist-test-slave-b83v"
server_key: "0ebf84bdb29ac329e7b317c4a759a02d"
server_key_iv: "678f9b7d722f94ffe871524b3f810ea5"
server_key_version: "encryptionkey@0"
I20260928 20:22:04.997730 16519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:22:05.010757 16519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:22:05.013428 16519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:22:05.013619 16519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:22:05.017745 16519 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.61:36447
I20260928 20:22:05.017779 16586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.61:36447 every 8 connection(s)
I20260928 20:22:05.018146 16519 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/minicluster-data/master-1/data/info.pb
I20260928 20:22:05.019069 16587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:22:05.020798 16587 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36979 } has no permanent_uuid. Determining permanent_uuid...
I20260928 20:22:05.025486 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 16519
I20260928 20:22:05.025624 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/minicluster-data/master-1/wal/instance
I20260928 20:22:05.025818 3144 external_mini_cluster.cc:1428] Setting key 2495ae9798b0e903cd993dee8d738a07
Sep 28 20:22:05 dist-test-slave-b83v krb5kdc[16431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626924, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.61@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:22:05.030982 16514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:05.021602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52116 (local address 127.3.18.62:36979)
0928 20:22:05.021661 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:05.021665 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:05.021798 (+ 133us) server_negotiation.cc:407] Connection header received
0928 20:22:05.022241 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:05.022242 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:05.022255 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:05.022321 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:22:05.023252 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:05.023896 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:05.024943 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:05.025150 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:05.025209 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:05.027881 (+ 2672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:22:05.027882 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:22:05.027883 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:22:05.027889 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:22:05.029525 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:05.029934 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:05.029935 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:05.029936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:05.030000 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:05.030256 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:05.030257 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:05.030258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:05.030379 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:22:05.030459 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:05.030792 (+ 333us) server_negotiation.cc:299] Negotiation successful
0928 20:22:05.030880 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 20:22:05.031165 16589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:05.021483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36979 (local address 127.0.0.1:52116)
0928 20:22:05.021665 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:05.021707 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 20:22:05.022157 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:22:05.022377 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:22:05.022405 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:22:05.022828 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:22:05.023142 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:05.023149 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:05.024005 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:05.024006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:05.024848 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:05.024850 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:05.025228 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:05.025229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:05.025397 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:05.026012 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:22:05.026022 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:22:05.027791 (+ 1769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:22:05.029619 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:22:05.029626 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:22:05.029628 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:22:05.029855 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:22:05.030069 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:22:05.030070 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:22:05.030070 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:22:05.030189 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:22:05.030450 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:22:05.030457 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:22:05.030659 (+ 202us) client_negotiation.cc:769] Sending connection context
0928 20:22:05.030760 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 20:22:05.030920 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":107,"threads_started":1}
I20260928 20:22:05.031749 16587 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.18.61" port: 36447 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.18.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.60@KRBTEST.COM" created.
Sep 28 20:22:05 dist-test-slave-b83v krb5kdc[16431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626924, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.61@KRBTEST.COM for kudu/127.3.18.61@KRBTEST.COM
I20260928 20:22:05.041195 16589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:05.032077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.61:36447 (local address 127.0.0.1:51922)
0928 20:22:05.032129 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:05.032139 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 20:22:05.032180 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:22:05.032421 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:22:05.032422 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:22:05.032567 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:22:05.032613 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:05.032615 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:05.033558 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:05.033559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:05.034100 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:05.034102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:05.034670 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:05.034671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:05.034737 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:05.035271 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:22:05.035277 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:22:05.037124 (+ 1847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:22:05.039629 (+ 2505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:22:05.039630 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:22:05.039631 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:22:05.039908 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:22:05.040151 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:22:05.040152 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:22:05.040152 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:22:05.040252 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:22:05.040740 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:22:05.040742 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:22:05.040804 (+ 62us) client_negotiation.cc:769] Sending connection context
0928 20:22:05.040896 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 20:22:05.041022 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 20:22:05.041209 16591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:05.032121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51922 (local address 127.3.18.61:36447)
0928 20:22:05.032280 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:05.032284 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:05.032312 (+ 28us) server_negotiation.cc:407] Connection header received
0928 20:22:05.032328 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:05.032329 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:05.032339 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:05.032414 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:22:05.032685 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:05.033470 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:05.034215 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:05.034582 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:05.034664 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:05.037236 (+ 2572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:22:05.037245 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:22:05.037245 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:22:05.037252 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:22:05.039530 (+ 2278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:05.040011 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:05.040015 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:05.040016 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:05.040080 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:05.040398 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:05.040399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:05.040399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:05.040582 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:22:05.040756 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:05.040929 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 20:22:05.041082 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":70,"server-negotiator.queue_time_us":133,"thread_start_us":88,"threads_started":1}
I20260928 20:22:05.041976 16587 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.18.60" port: 44445 } has no permanent_uuid. Determining permanent_uuid...
I20260928 20:22:05.042418 16589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:05.042264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.60:44445 (local address 127.0.0.1:39066)
0928 20:22:05.042318 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:05.042378 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.60:44445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 20:22:05.042632 16538 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.18.60:44445: connect: Connection refused (error 111)
I20260928 20:22:05.043148 16589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:05.043030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.60:44445 (local address 127.0.0.1:39080)
0928 20:22:05.043083 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:05.043117 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.60:44445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 20:22:05.043309 16587 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.18.60:44445: Network error: Client connection negotiation failed: client connection to 127.3.18.60:44445: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.60.keytab.
Entry for principal kudu/127.3.18.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.60.keytab.
I20260928 20:22:05.055886 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.60:44445
--webserver_interface=127.3.18.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.60.keytab
--principal=kudu/127.3.18.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:37461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.18.62:36979,127.3.18.61:36447,127.3.18.60:44445
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 20:22:05.074028 16512 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.18.61" port: 36447 } attempt: 3
Sep 28 20:22:05 dist-test-slave-b83v krb5kdc[16431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626924, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.62@KRBTEST.COM for kudu/127.3.18.61@KRBTEST.COM
I20260928 20:22:05.081605 16515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:05.074461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.61:36447 (local address 127.0.0.1:51924)
0928 20:22:05.074509 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:05.074528 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 20:22:05.074599 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:22:05.074767 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:22:05.074767 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:22:05.074929 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:22:05.075000 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:05.075003 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:05.075788 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:05.075789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:05.076328 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:05.076331 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:05.076736 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:05.076737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:05.076823 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:05.077344 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:22:05.077350 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:22:05.078690 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:22:05.080483 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:22:05.080484 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:22:05.080485 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:22:05.080666 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:22:05.080892 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:22:05.080893 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:22:05.080893 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:22:05.080958 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:22:05.081267 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:22:05.081268 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:22:05.081316 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 20:22:05.081398 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 20:22:05.081507 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260928 20:22:05.081657 16591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:05.074519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51924 (local address 127.3.18.61:36447)
0928 20:22:05.074587 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:05.074590 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:05.074601 (+ 11us) server_negotiation.cc:407] Connection header received
0928 20:22:05.074670 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:05.074671 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:05.074682 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:05.074747 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:22:05.075089 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:05.075665 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:05.076436 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:05.076658 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:05.076739 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:05.078787 (+ 2048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:22:05.078789 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:22:05.078789 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:22:05.078800 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:22:05.080381 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:05.080754 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:05.080755 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:05.080756 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:05.080817 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:05.081068 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:05.081069 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:05.081069 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:05.081191 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:22:05.081264 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:05.081423 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 20:22:05.081566 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 20:22:05.082659 16512 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.18.60" port: 44445 } has no permanent_uuid. Determining permanent_uuid...
I20260928 20:22:05.083076 16515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:05.082952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.60:44445 (local address 127.0.0.1:39084)
0928 20:22:05.083013 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:05.083045 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.60:44445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 20:22:05.083438 16515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:05.083359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.60:44445 (local address 127.0.0.1:39086)
0928 20:22:05.083400 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:05.083423 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.60:44445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 20:22:05.083539 16512 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.18.60:44445: Network error: Client connection negotiation failed: client connection to 127.3.18.60:44445: connect: Connection refused (error 111)
I20260928 20:22:05.092552 16587 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.18.60" port: 44445 } attempt: 1
I20260928 20:22:05.093127 16589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:05.092968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.60:44445 (local address 127.0.0.1:39088)
0928 20:22:05.093029 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:05.093084 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.60:44445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 20:22:05.093526 16589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:05.093423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.60:44445 (local address 127.0.0.1:39102)
0928 20:22:05.093470 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:05.093503 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.60:44445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 20:22:05.093632 16587 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.18.60:44445: Network error: Client connection negotiation failed: client connection to 127.3.18.60:44445: connect: Connection refused (error 111)
I20260928 20:22:05.114692 16512 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.18.60" port: 44445 } attempt: 1
I20260928 20:22:05.115259 16515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:05.115083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.60:44445 (local address 127.0.0.1:39106)
0928 20:22:05.115150 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:05.115193 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.60:44445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 20:22:05.115739 16515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:05.115588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.60:44445 (local address 127.0.0.1:39116)
0928 20:22:05.115660 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:05.115710 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.60:44445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 20:22:05.115924 16512 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.18.60:44445: Network error: Client connection negotiation failed: client connection to 127.3.18.60:44445: connect: Connection refused (error 111)
W20260928 20:22:05.143882 16594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:22:05.144134 16594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:22:05.144176 16594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:22:05.145347 16594 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:22:05.145431 16594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:22:05.145455 16594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:22:05.145475 16594 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:22:05.145503 16594 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:22:05.147099 16594 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:37461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.18.62:36979,127.3.18.61:36447,127.3.18.60:44445
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.60.keytab
--principal=kudu/127.3.18.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.60:44445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.18.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:22:05.147403 16594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:22:05.147646 16594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:22:05.153478 16599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:22:05.153471 16594 server_base.cc:1034] running on GCE node
W20260928 20:22:05.153432 16600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:22:05.153465 16602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:22:05.153889 16594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:22:05.154259 16594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:22:05.155519 16594 hybrid_clock.cc:648] HybridClock initialized: now 1790626925155438 us; error 43 us; skew 500 ppm
Sep 28 20:22:05 dist-test-slave-b83v krb5kdc[16431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626925, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:22:05.158556 16594 init.cc:377] Logged in from keytab as kudu/127.3.18.60@KRBTEST.COM (short username kudu)
I20260928 20:22:05.159126 16594 webserver.cc:460] Webserver started at http://127.3.18.60:41527/ using document root <none> and password file <none>
I20260928 20:22:05.159368 16594 fs_manager.cc:359] Metadata directory not provided
I20260928 20:22:05.159443 16594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:22:05.159591 16594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:22:05.160761 16594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/minicluster-data/master-2/data/instance:
uuid: "19cdf9b315d9418e9b5f7e33be59b683"
format_stamp: "Formatted at 2026-09-28 20:22:05 on dist-test-slave-b83v"
server_key: "23caeab16eeeac71e378a5716d93febe"
server_key_iv: "8cfc564e2103018acdd803c22c905d52"
server_key_version: "encryptionkey@0"
I20260928 20:22:05.161130 16594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/minicluster-data/master-2/wal/instance:
uuid: "19cdf9b315d9418e9b5f7e33be59b683"
format_stamp: "Formatted at 2026-09-28 20:22:05 on dist-test-slave-b83v"
server_key: "23caeab16eeeac71e378a5716d93febe"
server_key_iv: "8cfc564e2103018acdd803c22c905d52"
server_key_version: "encryptionkey@0"
I20260928 20:22:05.161783 16587 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.18.60" port: 44445 } attempt: 2
I20260928 20:22:05.162264 16589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:05.162110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.60:44445 (local address 127.0.0.1:39120)
0928 20:22:05.162178 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:05.162220 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.60:44445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 20:22:05.162731 16589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:05.162622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.60:44445 (local address 127.0.0.1:39124)
0928 20:22:05.162682 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:05.162709 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.60:44445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 20:22:05.162860 16587 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.18.60:44445: Network error: Client connection negotiation failed: client connection to 127.3.18.60:44445: connect: Connection refused (error 111)
I20260928 20:22:05.162891 16594 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 20:22:05.163895 16609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:22:05.164202 16594 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:22:05.164285 16594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/minicluster-data/master-2/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/minicluster-data/master-2/wal
uuid: "19cdf9b315d9418e9b5f7e33be59b683"
format_stamp: "Formatted at 2026-09-28 20:22:05 on dist-test-slave-b83v"
server_key: "23caeab16eeeac71e378a5716d93febe"
server_key_iv: "8cfc564e2103018acdd803c22c905d52"
server_key_version: "encryptionkey@0"
I20260928 20:22:05.164393 16594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:22:05.183686 16594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:22:05.186309 16594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:22:05.186436 16594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:22:05.191121 16512 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.18.60" port: 44445 } attempt: 2
I20260928 20:22:05.191418 16594 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.60:44445
I20260928 20:22:05.191462 16661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.60:44445 every 8 connection(s)
I20260928 20:22:05.191983 16594 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/minicluster-data/master-2/data/info.pb
I20260928 20:22:05.192868 16662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:22:05.195120 16662 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36979 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 20:22:05 dist-test-slave-b83v krb5kdc[16431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626924, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.62@KRBTEST.COM for kudu/127.3.18.60@KRBTEST.COM
Sep 28 20:22:05 dist-test-slave-b83v krb5kdc[16431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626925, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.60@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:22:05.201426 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 16594
I20260928 20:22:05.201553 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790626764250411-3144-0/minicluster-data/master-2/wal/instance
I20260928 20:22:05.201802 3144 external_mini_cluster.cc:1428] Setting key 09e0c09b44c4865bc9528f5b47b9d494
I20260928 20:22:05.202853 16515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:05.191465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.60:44445 (local address 127.0.0.1:39134)
0928 20:22:05.191532 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:05.191544 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:22:05.191592 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:22:05.193398 (+ 1806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:22:05.193399 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:22:05.193615 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:22:05.193686 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:05.193688 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:05.195414 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:05.195416 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:05.196250 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:05.196254 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:05.196974 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:05.196976 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:05.197034 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:05.197537 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:22:05.197543 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:22:05.198913 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:22:05.201338 (+ 2425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:22:05.201339 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:22:05.201340 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:22:05.201566 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:22:05.201896 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:22:05.201897 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:22:05.201897 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:22:05.201965 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:22:05.202456 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:22:05.202457 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:22:05.202508 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 20:22:05.202587 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 20:22:05.202724 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 20:22:05.203073 16663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:05.191786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39134 (local address 127.3.18.60:44445)
0928 20:22:05.192399 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:05.192411 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:05.192450 (+ 39us) server_negotiation.cc:407] Connection header received
0928 20:22:05.193114 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:05.193123 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:05.193211 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:05.193376 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:22:05.193800 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:05.195251 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:05.196373 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:05.196864 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:05.197010 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:05.199024 (+ 2014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:22:05.199033 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:22:05.199033 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:22:05.199046 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:22:05.201219 (+ 2173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:05.201711 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:05.201714 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:05.201714 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:05.201818 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:05.202041 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:05.202042 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:05.202043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:05.202350 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:22:05.202426 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:05.202680 (+ 254us) server_negotiation.cc:299] Negotiation successful
0928 20:22:05.202814 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":496,"threads_started":1}
I20260928 20:22:05.204861 16514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:05.196015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52132 (local address 127.3.18.62:36979)
0928 20:22:05.196057 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:05.196059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:05.196472 (+ 413us) server_negotiation.cc:407] Connection header received
0928 20:22:05.196495 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:05.196496 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:05.196514 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:05.196598 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:22:05.197126 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:05.197580 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:05.198639 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:05.198887 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:05.198950 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:05.201458 (+ 2508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:22:05.201460 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:22:05.201460 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:22:05.201468 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:22:05.203207 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:05.203976 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:05.203977 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:05.203978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:05.204041 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:05.204265 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:05.204266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:05.204266 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:05.204412 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:22:05.204495 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:05.204685 (+ 190us) server_negotiation.cc:299] Negotiation successful
0928 20:22:05.204769 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 20:22:05.204885 16665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:05.195963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36979 (local address 127.0.0.1:52132)
0928 20:22:05.196194 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:05.196239 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 20:22:05.196373 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:22:05.196666 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:22:05.196669 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:22:05.196942 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:22:05.197025 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:05.197027 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:05.197674 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:05.197675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:05.198524 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:05.198527 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:05.198985 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:05.198986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:05.199077 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:05.199641 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:22:05.199645 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:22:05.201333 (+ 1688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:22:05.203348 (+ 2015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:22:05.203353 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:22:05.203355 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:22:05.203587 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:22:05.204115 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:22:05.204116 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:22:05.204116 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:22:05.204177 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:22:05.204487 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:22:05.204493 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:22:05.204547 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 20:22:05.204656 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 20:22:05.204769 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":153,"threads_started":1}
I20260928 20:22:05.205380 16662 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.18.61" port: 36447 } has no permanent_uuid. Determining permanent_uuid...
I20260928 20:22:05.205698 16512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d73bade8df3436da2af29916e688cd0: Bootstrap starting.
I20260928 20:22:05.206492 16512 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d73bade8df3436da2af29916e688cd0: Neither blocks nor log segments found. Creating new log.
I20260928 20:22:05.206868 16512 log.cc:824] T 00000000000000000000000000000000 P 0d73bade8df3436da2af29916e688cd0: Log is configured to *not* fsync() on all Append() calls
I20260928 20:22:05.207734 16512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d73bade8df3436da2af29916e688cd0: No bootstrap required, opened a new log
Sep 28 20:22:05 dist-test-slave-b83v krb5kdc[16431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.18.60@KRBTEST.COM
I20260928 20:22:05.209817 16512 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d73bade8df3436da2af29916e688cd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d73bade8df3436da2af29916e688cd0" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36979 } } peers { permanent_uuid: "5e39f28d7fb349c795d564741dc504fb" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 36447 } } peers { permanent_uuid: "19cdf9b315d9418e9b5f7e33be59b683" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 44445 } }
I20260928 20:22:05.209977 16512 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d73bade8df3436da2af29916e688cd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:22:05.210024 16512 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d73bade8df3436da2af29916e688cd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d73bade8df3436da2af29916e688cd0, State: Initialized, Role: FOLLOWER
I20260928 20:22:05.210192 16512 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d73bade8df3436da2af29916e688cd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d73bade8df3436da2af29916e688cd0" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36979 } } peers { permanent_uuid: "5e39f28d7fb349c795d564741dc504fb" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 36447 } } peers { permanent_uuid: "19cdf9b315d9418e9b5f7e33be59b683" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 44445 } }
I20260928 20:22:05.210536 16512 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d73bade8df3436da2af29916e688cd0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:22:05.211645 16669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d73bade8df3436da2af29916e688cd0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d73bade8df3436da2af29916e688cd0" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36979 } } peers { permanent_uuid: "5e39f28d7fb349c795d564741dc504fb" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 36447 } } peers { permanent_uuid: "19cdf9b315d9418e9b5f7e33be59b683" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 44445 } } }
Sep 28 20:22:05 dist-test-slave-b83v krb5kdc[16431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626925, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.60@KRBTEST.COM for kudu/127.3.18.61@KRBTEST.COM
I20260928 20:22:05.211841 16669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d73bade8df3436da2af29916e688cd0 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 20:22:05.212917 16667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:05.202370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39150 (local address 127.3.18.60:44445)
0928 20:22:05.202720 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:05.202723 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:05.202729 (+ 6us) server_negotiation.cc:407] Connection header received
0928 20:22:05.202747 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:05.202748 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:05.202758 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:05.202813 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:22:05.203916 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:05.204613 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:05.206270 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:05.206473 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:05.206531 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:05.209341 (+ 2810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:22:05.209343 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:22:05.209343 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:22:05.209351 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:22:05.211311 (+ 1960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:05.211835 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:05.211835 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:05.211836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:05.211904 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:05.212163 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:05.212164 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:05.212164 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:05.212316 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:22:05.212495 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:05.212596 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 20:22:05.212698 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":108,"threads_started":1}
W20260928 20:22:05.212721 16680 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0d73bade8df3436da2af29916e688cd0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 20:22:05.213440 16680 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 20:22:05.215394 16665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:05.205682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.61:36447 (local address 127.0.0.1:51936)
0928 20:22:05.205724 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:05.205733 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 20:22:05.205766 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:22:05.207390 (+ 1624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:22:05.207392 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:22:05.207562 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:22:05.207608 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:05.207610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:05.208529 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:05.208530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:05.209366 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:05.209369 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:05.209821 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:05.209822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:05.209886 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:05.210787 (+ 901us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:22:05.210791 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:22:05.212195 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:22:05.214301 (+ 2106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:22:05.214302 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:22:05.214302 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:22:05.214505 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:22:05.214751 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:22:05.214752 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:22:05.214752 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:22:05.214810 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:22:05.215092 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:22:05.215093 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:22:05.215136 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 20:22:05.215199 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 20:22:05.215297 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260928 20:22:05.215696 16591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:05.207177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51936 (local address 127.3.18.61:36447)
0928 20:22:05.207230 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:05.207233 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:05.207241 (+ 8us) server_negotiation.cc:407] Connection header received
0928 20:22:05.207267 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:05.207268 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:05.207277 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:05.207335 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:22:05.207705 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:05.208430 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:05.209460 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:05.209738 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:05.209808 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:05.212330 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:22:05.212332 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:22:05.212332 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:22:05.212340 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:22:05.214208 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:05.214603 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:05.214604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:05.214604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:05.214672 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:05.214874 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:05.214875 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:05.214875 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:05.215002 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:22:05.215063 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:05.215505 (+ 442us) server_negotiation.cc:299] Negotiation successful
0928 20:22:05.215599 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 20:22:05.214571 16444 master_runner.cc:186] Error getting master registration for 127.3.18.61:36447: OK, instance_id { permanent_uuid: "5e39f28d7fb349c795d564741dc504fb" instance_seqno: 1790626925016622 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 20:22:05.217489 16444 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 20:22:05.217898 16662 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.18.60" port: 44445 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 20:22:05 dist-test-slave-b83v krb5kdc[16431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
Sep 28 20:22:05 dist-test-slave-b83v krb5kdc[16431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626925, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.60@KRBTEST.COM for kudu/127.3.18.60@KRBTEST.COM
I20260928 20:22:05.224027 16514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:05.213599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52134 (local address 127.3.18.62:36979)
0928 20:22:05.213644 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:05.213646 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:05.213761 (+ 115us) server_negotiation.cc:407] Connection header received
0928 20:22:05.213782 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:05.213783 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:05.213792 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:05.213834 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:22:05.214484 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:05.215230 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:05.216703 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:05.217274 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:05.217363 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:05.220346 (+ 2983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:22:05.220347 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:22:05.220348 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:22:05.220356 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:22:05.222229 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:05.222639 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:05.222641 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:05.222641 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:05.222700 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:05.223236 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:05.223238 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:05.223238 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:05.223384 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:22:05.223569 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:05.223834 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 20:22:05.223924 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 20:22:05.225770 16665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:05.218193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.60:44445 (local address 127.0.0.1:39154)
0928 20:22:05.218252 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:05.218264 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:22:05.218305 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:22:05.218437 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:22:05.218437 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:22:05.218611 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:22:05.218654 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:05.218657 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:05.219491 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:05.219492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:05.220113 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:05.220116 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:05.220695 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:05.220696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:05.220742 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:05.221183 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:22:05.221188 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:22:05.222494 (+ 1306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:22:05.224716 (+ 2222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:22:05.224717 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:22:05.224717 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:22:05.224927 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:22:05.225127 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:22:05.225128 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:22:05.225128 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:22:05.225185 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:22:05.225434 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:22:05.225434 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:22:05.225471 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 20:22:05.225540 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 20:22:05.225656 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 20:22:05.225816 16667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:05.218247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39154 (local address 127.3.18.60:44445)
0928 20:22:05.218284 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:05.218286 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:05.218295 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:22:05.218359 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:05.218360 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:05.218372 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:05.218435 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:22:05.218740 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:05.219395 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:05.220333 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:05.220620 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:05.220701 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:05.222613 (+ 1912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:22:05.222614 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:22:05.222614 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:22:05.222620 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:22:05.224634 (+ 2014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:05.224999 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:05.224999 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:05.224999 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:05.225057 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:05.225251 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:05.225253 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:05.225253 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:05.225368 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:22:05.225442 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:05.225567 (+ 125us) server_negotiation.cc:299] Negotiation successful
0928 20:22:05.225708 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":70,"server-negotiator.queue_time_us":18}
I20260928 20:22:05.227630 16662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19cdf9b315d9418e9b5f7e33be59b683: Bootstrap starting.
I20260928 20:22:05.228590 16662 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19cdf9b315d9418e9b5f7e33be59b683: Neither blocks nor log segments found. Creating new log.
I20260928 20:22:05.228955 16662 log.cc:824] T 00000000000000000000000000000000 P 19cdf9b315d9418e9b5f7e33be59b683: Log is configured to *not* fsync() on all Append() calls
I20260928 20:22:05.229838 16662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19cdf9b315d9418e9b5f7e33be59b683: No bootstrap required, opened a new log
Sep 28 20:22:05 dist-test-slave-b83v krb5kdc[16431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.18.61@KRBTEST.COM
I20260928 20:22:05.231487 16662 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19cdf9b315d9418e9b5f7e33be59b683 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d73bade8df3436da2af29916e688cd0" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36979 } } peers { permanent_uuid: "5e39f28d7fb349c795d564741dc504fb" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 36447 } } peers { permanent_uuid: "19cdf9b315d9418e9b5f7e33be59b683" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 44445 } }
I20260928 20:22:05.231626 16662 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19cdf9b315d9418e9b5f7e33be59b683 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:22:05.231652 16662 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19cdf9b315d9418e9b5f7e33be59b683 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19cdf9b315d9418e9b5f7e33be59b683, State: Initialized, Role: FOLLOWER
I20260928 20:22:05.231823 16662 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19cdf9b315d9418e9b5f7e33be59b683 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d73bade8df3436da2af29916e688cd0" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36979 } } peers { permanent_uuid: "5e39f28d7fb349c795d564741dc504fb" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 36447 } } peers { permanent_uuid: "19cdf9b315d9418e9b5f7e33be59b683" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 44445 } }
I20260928 20:22:05.232112 16683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19cdf9b315d9418e9b5f7e33be59b683 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d73bade8df3436da2af29916e688cd0" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36979 } } peers { permanent_uuid: "5e39f28d7fb349c795d564741dc504fb" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 36447 } } peers { permanent_uuid: "19cdf9b315d9418e9b5f7e33be59b683" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 44445 } } }
I20260928 20:22:05.232185 16662 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19cdf9b315d9418e9b5f7e33be59b683 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:22:05.232229 16683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19cdf9b315d9418e9b5f7e33be59b683 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 20:22:05.234093 16591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:05.224772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51942 (local address 127.3.18.61:36447)
0928 20:22:05.224821 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:05.224823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:05.224831 (+ 8us) server_negotiation.cc:407] Connection header received
0928 20:22:05.224858 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:05.224859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:05.224870 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:05.224939 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:22:05.225595 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:05.226324 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:05.227174 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:05.227356 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:05.227412 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:05.230466 (+ 3054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:22:05.230468 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:22:05.230468 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:22:05.230478 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:22:05.232306 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:05.232730 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:05.232732 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:05.232732 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:05.232801 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:05.233024 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:05.233025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:05.233026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:05.233164 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:22:05.233864 (+ 700us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:05.233908 (+ 44us) server_negotiation.cc:299] Negotiation successful
0928 20:22:05.233996 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
W20260928 20:22:05.234228 16694 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 19cdf9b315d9418e9b5f7e33be59b683: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 20:22:05.234285 16694 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 20:22:05.235721 3144 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 20:22:05.235865 16594 master_runner.cc:186] Error getting master registration for 127.3.18.61:36447: OK, instance_id { permanent_uuid: "5e39f28d7fb349c795d564741dc504fb" instance_seqno: 1790626925016622 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 20:22:05.238308 16594 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 20:22:05.253919 16667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:05.240499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39160 (local address 127.3.18.60:44445)
0928 20:22:05.240549 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:05.240551 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:05.244246 (+ 3695us) server_negotiation.cc:407] Connection header received
0928 20:22:05.244319 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:05.244320 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:05.244332 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:05.244400 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:22:05.245148 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:05.245899 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:05.247068 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:05.247395 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:05.247437 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:05.249570 (+ 2133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:22:05.249572 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:22:05.249572 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:22:05.249580 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:22:05.251365 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:05.252541 (+ 1176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:05.252542 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:05.252543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:05.252627 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:05.252876 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:05.252877 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:05.252877 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:05.253024 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:22:05.253134 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:05.253698 (+ 564us) server_negotiation.cc:299] Negotiation successful
0928 20:22:05.253806 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 20:22:05.253986 16587 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.18.60" port: 44445 } attempt: 3
I20260928 20:22:05.255625 16514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:05.240223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52148 (local address 127.3.18.62:36979)
0928 20:22:05.240285 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:05.240288 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:05.244047 (+ 3759us) server_negotiation.cc:407] Connection header received
0928 20:22:05.244144 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:05.244147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:05.244162 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:05.244195 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:22:05.246695 (+ 2500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:05.247443 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:05.248601 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:05.248907 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:05.248949 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:05.251890 (+ 2941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:22:05.251891 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:22:05.251892 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:22:05.251900 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:22:05.253813 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:05.254742 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:05.254744 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:05.254744 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:05.254811 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:05.255026 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:05.255028 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:05.255028 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:05.255164 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:22:05.255244 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:05.255429 (+ 185us) server_negotiation.cc:299] Negotiation successful
0928 20:22:05.255514 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 20:22:05.256963 16591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:05.240726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51952 (local address 127.3.18.61:36447)
0928 20:22:05.240779 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:05.240781 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:05.244425 (+ 3644us) server_negotiation.cc:407] Connection header received
0928 20:22:05.244537 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:05.244539 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:05.244551 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:05.244583 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:22:05.245247 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:05.245978 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:05.249619 (+ 3641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:05.249928 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:05.249974 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:05.252877 (+ 2903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:22:05.252878 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:22:05.252879 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:22:05.252888 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:22:05.255072 (+ 2184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:05.255565 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:05.255566 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:05.255567 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:05.255626 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:05.255860 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:05.255861 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:05.255861 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:05.255982 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:22:05.256550 (+ 568us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:05.256790 (+ 240us) server_negotiation.cc:299] Negotiation successful
0928 20:22:05.256872 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
Sep 28 20:22:05 dist-test-slave-b83v krb5kdc[16431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626924, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.61@KRBTEST.COM for kudu/127.3.18.60@KRBTEST.COM
I20260928 20:22:05.261752 16667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:05.254321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39164 (local address 127.3.18.60:44445)
0928 20:22:05.254361 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:05.254363 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:05.254371 (+ 8us) server_negotiation.cc:407] Connection header received
0928 20:22:05.254439 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:05.254440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:05.254450 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:05.254512 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:22:05.255134 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:05.255841 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:05.256794 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:05.257087 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:05.257160 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:05.259142 (+ 1982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:22:05.259143 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:22:05.259144 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:22:05.259150 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:22:05.260617 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:05.260977 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:05.260978 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:05.260979 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:05.261037 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:05.261237 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:05.261238 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:05.261239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:05.261346 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:22:05.261420 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:05.261597 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 20:22:05.261669 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 20:22:05.261749 16589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:05.254227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.60:44445 (local address 127.0.0.1:39164)
0928 20:22:05.254327 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:05.254339 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:22:05.254379 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:22:05.254558 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:22:05.254559 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:22:05.254769 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:22:05.254820 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:05.254822 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:05.255939 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:05.255941 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:05.256680 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:05.256683 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:05.257210 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:05.257211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:05.257291 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:05.257704 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:22:05.257708 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:22:05.259056 (+ 1348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:22:05.260723 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:22:05.260724 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:22:05.260725 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:22:05.260893 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:22:05.261105 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:22:05.261106 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:22:05.261107 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:22:05.261165 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:22:05.261413 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:22:05.261414 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:22:05.261459 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 20:22:05.261548 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 20:22:05.261650 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260928 20:22:05.263135 16587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e39f28d7fb349c795d564741dc504fb: Bootstrap starting.
I20260928 20:22:05.263911 16587 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e39f28d7fb349c795d564741dc504fb: Neither blocks nor log segments found. Creating new log.
I20260928 20:22:05.264271 16587 log.cc:824] T 00000000000000000000000000000000 P 5e39f28d7fb349c795d564741dc504fb: Log is configured to *not* fsync() on all Append() calls
I20260928 20:22:05.265190 16587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e39f28d7fb349c795d564741dc504fb: No bootstrap required, opened a new log
I20260928 20:22:05.267373 16587 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e39f28d7fb349c795d564741dc504fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d73bade8df3436da2af29916e688cd0" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36979 } } peers { permanent_uuid: "5e39f28d7fb349c795d564741dc504fb" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 36447 } } peers { permanent_uuid: "19cdf9b315d9418e9b5f7e33be59b683" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 44445 } }
I20260928 20:22:05.267501 16587 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e39f28d7fb349c795d564741dc504fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:22:05.267553 16587 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e39f28d7fb349c795d564741dc504fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e39f28d7fb349c795d564741dc504fb, State: Initialized, Role: FOLLOWER
I20260928 20:22:05.267686 16587 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e39f28d7fb349c795d564741dc504fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d73bade8df3436da2af29916e688cd0" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36979 } } peers { permanent_uuid: "5e39f28d7fb349c795d564741dc504fb" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 36447 } } peers { permanent_uuid: "19cdf9b315d9418e9b5f7e33be59b683" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 44445 } }
I20260928 20:22:05.268062 16704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e39f28d7fb349c795d564741dc504fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d73bade8df3436da2af29916e688cd0" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36979 } } peers { permanent_uuid: "5e39f28d7fb349c795d564741dc504fb" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 36447 } } peers { permanent_uuid: "19cdf9b315d9418e9b5f7e33be59b683" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 44445 } } }
I20260928 20:22:05.268147 16587 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e39f28d7fb349c795d564741dc504fb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:22:05.268174 16704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e39f28d7fb349c795d564741dc504fb [sys.catalog]: This master's current role is: FOLLOWER
W20260928 20:22:05.270673 16715 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5e39f28d7fb349c795d564741dc504fb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 20:22:05.270740 16715 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 20:22:05.272392 16519 master_runner.cc:263] No leader master found from master 5e39f28d7fb349c795d564741dc504fb
I20260928 20:22:05.272478 16519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 20:22:05.341359 16465 tablet.cc:2369] T 00000000000000000000000000000000 P 0d73bade8df3436da2af29916e688cd0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 20:22:05.376803 16704 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e39f28d7fb349c795d564741dc504fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:22:05.376940 16704 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e39f28d7fb349c795d564741dc504fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d73bade8df3436da2af29916e688cd0" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36979 } } peers { permanent_uuid: "5e39f28d7fb349c795d564741dc504fb" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 36447 } } peers { permanent_uuid: "19cdf9b315d9418e9b5f7e33be59b683" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 44445 } }
I20260928 20:22:05.377192 16704 leader_election.cc:288] T 00000000000000000000000000000000 P 5e39f28d7fb349c795d564741dc504fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d73bade8df3436da2af29916e688cd0 (127.3.18.62:36979), 19cdf9b315d9418e9b5f7e33be59b683 (127.3.18.60:44445)
I20260928 20:22:05.377712 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5e39f28d7fb349c795d564741dc504fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19cdf9b315d9418e9b5f7e33be59b683" is_pre_election: true
I20260928 20:22:05.377882 16636 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 19cdf9b315d9418e9b5f7e33be59b683 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e39f28d7fb349c795d564741dc504fb in term 0.
I20260928 20:22:05.378211 16538 leader_election.cc:302] T 00000000000000000000000000000000 P 5e39f28d7fb349c795d564741dc504fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19cdf9b315d9418e9b5f7e33be59b683, 5e39f28d7fb349c795d564741dc504fb; no voters:
I20260928 20:22:05.378393 16704 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e39f28d7fb349c795d564741dc504fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 20:22:05.378444 16704 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e39f28d7fb349c795d564741dc504fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 20:22:05.378465 16704 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e39f28d7fb349c795d564741dc504fb [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:22:05.379415 16704 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e39f28d7fb349c795d564741dc504fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d73bade8df3436da2af29916e688cd0" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36979 } } peers { permanent_uuid: "5e39f28d7fb349c795d564741dc504fb" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 36447 } } peers { permanent_uuid: "19cdf9b315d9418e9b5f7e33be59b683" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 44445 } }
I20260928 20:22:05.379552 16704 leader_election.cc:288] T 00000000000000000000000000000000 P 5e39f28d7fb349c795d564741dc504fb [CANDIDATE]: Term 1 election: Requested vote from peers 0d73bade8df3436da2af29916e688cd0 (127.3.18.62:36979), 19cdf9b315d9418e9b5f7e33be59b683 (127.3.18.60:44445)
I20260928 20:22:05.379913 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5e39f28d7fb349c795d564741dc504fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19cdf9b315d9418e9b5f7e33be59b683"
I20260928 20:22:05.379998 16636 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19cdf9b315d9418e9b5f7e33be59b683 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:22:05.380872 16636 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 19cdf9b315d9418e9b5f7e33be59b683 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e39f28d7fb349c795d564741dc504fb in term 1.
I20260928 20:22:05.381119 16538 leader_election.cc:302] T 00000000000000000000000000000000 P 5e39f28d7fb349c795d564741dc504fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19cdf9b315d9418e9b5f7e33be59b683, 5e39f28d7fb349c795d564741dc504fb; no voters:
I20260928 20:22:05.381246 16704 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e39f28d7fb349c795d564741dc504fb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:22:05.381402 16704 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e39f28d7fb349c795d564741dc504fb [term 1 LEADER]: Becoming Leader. State: Replica: 5e39f28d7fb349c795d564741dc504fb, State: Running, Role: LEADER
I20260928 20:22:05.381494 16704 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e39f28d7fb349c795d564741dc504fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d73bade8df3436da2af29916e688cd0" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36979 } } peers { permanent_uuid: "5e39f28d7fb349c795d564741dc504fb" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 36447 } } peers { permanent_uuid: "19cdf9b315d9418e9b5f7e33be59b683" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 44445 } }
I20260928 20:22:05.381907 16704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e39f28d7fb349c795d564741dc504fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e39f28d7fb349c795d564741dc504fb. Latest consensus state: current_term: 1 leader_uuid: "5e39f28d7fb349c795d564741dc504fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d73bade8df3436da2af29916e688cd0" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36979 } } peers { permanent_uuid: "5e39f28d7fb349c795d564741dc504fb" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 36447 } } peers { permanent_uuid: "19cdf9b315d9418e9b5f7e33be59b683" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 44445 } } }
I20260928 20:22:05.381974 16704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e39f28d7fb349c795d564741dc504fb [sys.catalog]: This master's current role is: LEADER
I20260928 20:22:05.382542 16485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5e39f28d7fb349c795d564741dc504fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d73bade8df3436da2af29916e688cd0"
I20260928 20:22:05.382609 16486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5e39f28d7fb349c795d564741dc504fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d73bade8df3436da2af29916e688cd0" is_pre_election: true
I20260928 20:22:05.382709 16485 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d73bade8df3436da2af29916e688cd0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:22:05.383647 16485 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0d73bade8df3436da2af29916e688cd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e39f28d7fb349c795d564741dc504fb in term 1.
I20260928 20:22:05.385242 16719 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:22:05.385481 16719 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:22:05.387209 16636 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 19cdf9b315d9418e9b5f7e33be59b683 [term 1 FOLLOWER]: Refusing update from remote peer 5e39f28d7fb349c795d564741dc504fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:22:05.387609 16704 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5e39f28d7fb349c795d564741dc504fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "19cdf9b315d9418e9b5f7e33be59b683" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 44445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:22:05.389341 16683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19cdf9b315d9418e9b5f7e33be59b683 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e39f28d7fb349c795d564741dc504fb. Latest consensus state: current_term: 1 leader_uuid: "5e39f28d7fb349c795d564741dc504fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d73bade8df3436da2af29916e688cd0" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36979 } } peers { permanent_uuid: "5e39f28d7fb349c795d564741dc504fb" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 36447 } } peers { permanent_uuid: "19cdf9b315d9418e9b5f7e33be59b683" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 44445 } } }
I20260928 20:22:05.389436 16683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19cdf9b315d9418e9b5f7e33be59b683 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 20:22:05.389938 16485 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0d73bade8df3436da2af29916e688cd0 [term 1 FOLLOWER]: Refusing update from remote peer 5e39f28d7fb349c795d564741dc504fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:22:05.390208 16704 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5e39f28d7fb349c795d564741dc504fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d73bade8df3436da2af29916e688cd0" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:22:05.391448 16669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d73bade8df3436da2af29916e688cd0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e39f28d7fb349c795d564741dc504fb. Latest consensus state: current_term: 1 leader_uuid: "5e39f28d7fb349c795d564741dc504fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d73bade8df3436da2af29916e688cd0" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36979 } } peers { permanent_uuid: "5e39f28d7fb349c795d564741dc504fb" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 36447 } } peers { permanent_uuid: "19cdf9b315d9418e9b5f7e33be59b683" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 44445 } } }
I20260928 20:22:05.391559 16669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d73bade8df3436da2af29916e688cd0 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 20:22:05.392740 16717 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e39f28d7fb349c795d564741dc504fb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5e39f28d7fb349c795d564741dc504fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d73bade8df3436da2af29916e688cd0" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36979 } } peers { permanent_uuid: "5e39f28d7fb349c795d564741dc504fb" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 36447 } } peers { permanent_uuid: "19cdf9b315d9418e9b5f7e33be59b683" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 44445 } } }
I20260928 20:22:05.392828 16717 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e39f28d7fb349c795d564741dc504fb [sys.catalog]: This master's current role is: LEADER
I20260928 20:22:05.393287 16669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d73bade8df3436da2af29916e688cd0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5e39f28d7fb349c795d564741dc504fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d73bade8df3436da2af29916e688cd0" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36979 } } peers { permanent_uuid: "5e39f28d7fb349c795d564741dc504fb" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 36447 } } peers { permanent_uuid: "19cdf9b315d9418e9b5f7e33be59b683" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 44445 } } }
I20260928 20:22:05.393368 16669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d73bade8df3436da2af29916e688cd0 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 20:22:05.393666 16720 mvcc.cc:205] Tried to move back new op lower bound from 7334407886383374336 to 7334407886363242496. Current Snapshot: MvccSnapshot[applied={T|T < 7334407886383374336}]
I20260928 20:22:05.394622 16704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e39f28d7fb349c795d564741dc504fb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5e39f28d7fb349c795d564741dc504fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d73bade8df3436da2af29916e688cd0" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36979 } } peers { permanent_uuid: "5e39f28d7fb349c795d564741dc504fb" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 36447 } } peers { permanent_uuid: "19cdf9b315d9418e9b5f7e33be59b683" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 44445 } } }
I20260928 20:22:05.394708 16704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e39f28d7fb349c795d564741dc504fb [sys.catalog]: This master's current role is: LEADER
I20260928 20:22:05.395150 16683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19cdf9b315d9418e9b5f7e33be59b683 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5e39f28d7fb349c795d564741dc504fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d73bade8df3436da2af29916e688cd0" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36979 } } peers { permanent_uuid: "5e39f28d7fb349c795d564741dc504fb" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 36447 } } peers { permanent_uuid: "19cdf9b315d9418e9b5f7e33be59b683" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 44445 } } }
I20260928 20:22:05.395238 16683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19cdf9b315d9418e9b5f7e33be59b683 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 20:22:05.395983 16721 mvcc.cc:205] Tried to move back new op lower bound from 7334407886383374336 to 7334407886363242496. Current Snapshot: MvccSnapshot[applied={T|T < 7334407886383374336}]
I20260928 20:22:05.396268 16719 catalog_manager.cc:1347] Generated new cluster ID: ff901dcb8e824822a059e995c1c5ba03
I20260928 20:22:05.396314 16719 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:22:05.400223 16723 mvcc.cc:205] Tried to move back new op lower bound from 7334407886383374336 to 7334407886363242496. Current Snapshot: MvccSnapshot[applied={T|T < 7334407886383374336}]
I20260928 20:22:05.414368 16719 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:22:05.415190 16719 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:22:05.424432 16719 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e39f28d7fb349c795d564741dc504fb: Generated new TSK 0
I20260928 20:22:05.424649 16719 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 20:22:06.213946 16680 catalog_manager.cc:1259] Loaded cluster ID: ff901dcb8e824822a059e995c1c5ba03
I20260928 20:22:06.214035 16680 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0d73bade8df3436da2af29916e688cd0: loading cluster ID for follower catalog manager: success
I20260928 20:22:06.214826 16680 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0d73bade8df3436da2af29916e688cd0: acquiring CA information for follower catalog manager: success
I20260928 20:22:06.215101 16680 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0d73bade8df3436da2af29916e688cd0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 20:22:06.234771 16694 catalog_manager.cc:1259] Loaded cluster ID: ff901dcb8e824822a059e995c1c5ba03
I20260928 20:22:06.234864 16694 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 19cdf9b315d9418e9b5f7e33be59b683: loading cluster ID for follower catalog manager: success
I20260928 20:22:06.235889 16694 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 19cdf9b315d9418e9b5f7e33be59b683: acquiring CA information for follower catalog manager: success
I20260928 20:22:06.236372 16694 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 19cdf9b315d9418e9b5f7e33be59b683: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 20:22:06.487332 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 16444
I20260928 20:22:06.492964 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 16519
I20260928 20:22:06.498045 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 16594
2026-09-28T20:22:06Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2988 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790626764250411-3144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:22:06 dist-test-slave-b83v krb5kdc[16734](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:22:06 dist-test-slave-b83v krb5kdc[16734](info): set up 2 sockets
Sep 28 20:22:06 dist-test-slave-b83v krb5kdc[16734](info): commencing operation
krb5kdc: starting...
W20260928 20:22:07.544140 3144 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Sep 28 20:22:07 dist-test-slave-b83v krb5kdc[16734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:22:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:22:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.18.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab.
Entry for principal kudu/127.3.18.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab.
I20260928 20:22:07.671975 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790626764250411-3144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790626764250411-3144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790626764250411-3144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:35099
--webserver_interface=127.3.18.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:38207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.18.62:35099
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:22:07.758595 16747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:22:07.758837 16747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:22:07.758867 16747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:22:07.760174 16747 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:22:07.760229 16747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:22:07.760246 16747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:22:07.760262 16747 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:22:07.760274 16747 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:22:07.761837 16747 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:38207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790626764250411-3144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790626764250411-3144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.18.62:35099
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:35099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.18.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790626764250411-3144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:22:07.762132 16747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:22:07.762403 16747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:22:07.767658 16752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:22:07.767742 16755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:22:07.767753 16753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:22:07.768211 16747 server_base.cc:1034] running on GCE node
I20260928 20:22:07.768497 16747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:22:07.768839 16747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:22:07.770053 16747 hybrid_clock.cc:648] HybridClock initialized: now 1790626927770011 us; error 52 us; skew 500 ppm
Sep 28 20:22:07 dist-test-slave-b83v krb5kdc[16734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626927, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:22:07.772755 16747 init.cc:377] Logged in from keytab as kudu/127.3.18.62@KRBTEST.COM (short username kudu)
I20260928 20:22:07.773319 16747 webserver.cc:460] Webserver started at http://127.3.18.62:43273/ using document root <none> and password file <none>
I20260928 20:22:07.773581 16747 fs_manager.cc:359] Metadata directory not provided
I20260928 20:22:07.773658 16747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:22:07.773819 16747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:22:07.775002 16747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790626764250411-3144-0/minicluster-data/master-0/data/instance:
uuid: "b1f87d7f7fc641ca9e1e8ba00d2110d0"
format_stamp: "Formatted at 2026-09-28 20:22:07 on dist-test-slave-b83v"
server_key: "9e9759dcc8164c5b911e6017f3a94596"
server_key_iv: "d30a6fad1bd306a8046e6fdf59a99d04"
server_key_version: "encryptionkey@0"
I20260928 20:22:07.775400 16747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790626764250411-3144-0/minicluster-data/master-0/wal/instance:
uuid: "b1f87d7f7fc641ca9e1e8ba00d2110d0"
format_stamp: "Formatted at 2026-09-28 20:22:07 on dist-test-slave-b83v"
server_key: "9e9759dcc8164c5b911e6017f3a94596"
server_key_iv: "d30a6fad1bd306a8046e6fdf59a99d04"
server_key_version: "encryptionkey@0"
I20260928 20:22:07.777029 16747 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 20:22:07.777997 16762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:22:07.778254 16747 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:22:07.778367 16747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790626764250411-3144-0/minicluster-data/master-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790626764250411-3144-0/minicluster-data/master-0/wal
uuid: "b1f87d7f7fc641ca9e1e8ba00d2110d0"
format_stamp: "Formatted at 2026-09-28 20:22:07 on dist-test-slave-b83v"
server_key: "9e9759dcc8164c5b911e6017f3a94596"
server_key_iv: "d30a6fad1bd306a8046e6fdf59a99d04"
server_key_version: "encryptionkey@0"
I20260928 20:22:07.778455 16747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790626764250411-3144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790626764250411-3144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790626764250411-3144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:22:07.789889 16747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:22:07.793151 16747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:22:07.793340 16747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:22:07.797569 16747 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.62:35099
I20260928 20:22:07.797638 16814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.62:35099 every 8 connection(s)
I20260928 20:22:07.798013 16747 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
I20260928 20:22:07.799008 16815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:22:07.801564 16815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1f87d7f7fc641ca9e1e8ba00d2110d0: Bootstrap starting.
I20260928 20:22:07.802192 16815 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1f87d7f7fc641ca9e1e8ba00d2110d0: Neither blocks nor log segments found. Creating new log.
I20260928 20:22:07.802506 16815 log.cc:824] T 00000000000000000000000000000000 P b1f87d7f7fc641ca9e1e8ba00d2110d0: Log is configured to *not* fsync() on all Append() calls
I20260928 20:22:07.803370 16815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1f87d7f7fc641ca9e1e8ba00d2110d0: No bootstrap required, opened a new log
I20260928 20:22:07.805186 16815 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1f87d7f7fc641ca9e1e8ba00d2110d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1f87d7f7fc641ca9e1e8ba00d2110d0" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 35099 } }
I20260928 20:22:07.805339 16815 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1f87d7f7fc641ca9e1e8ba00d2110d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:22:07.805362 16815 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1f87d7f7fc641ca9e1e8ba00d2110d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1f87d7f7fc641ca9e1e8ba00d2110d0, State: Initialized, Role: FOLLOWER
I20260928 20:22:07.805536 16815 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1f87d7f7fc641ca9e1e8ba00d2110d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1f87d7f7fc641ca9e1e8ba00d2110d0" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 35099 } }
I20260928 20:22:07.805624 16815 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1f87d7f7fc641ca9e1e8ba00d2110d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:22:07.805645 16815 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1f87d7f7fc641ca9e1e8ba00d2110d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:22:07.805725 16815 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1f87d7f7fc641ca9e1e8ba00d2110d0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:22:07.806291 16815 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1f87d7f7fc641ca9e1e8ba00d2110d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1f87d7f7fc641ca9e1e8ba00d2110d0" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 35099 } }
I20260928 20:22:07.806402 16815 leader_election.cc:302] T 00000000000000000000000000000000 P b1f87d7f7fc641ca9e1e8ba00d2110d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1f87d7f7fc641ca9e1e8ba00d2110d0; no voters:
I20260928 20:22:07.806620 16815 leader_election.cc:288] T 00000000000000000000000000000000 P b1f87d7f7fc641ca9e1e8ba00d2110d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:22:07.806685 16818 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1f87d7f7fc641ca9e1e8ba00d2110d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:22:07.806903 16818 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1f87d7f7fc641ca9e1e8ba00d2110d0 [term 1 LEADER]: Becoming Leader. State: Replica: b1f87d7f7fc641ca9e1e8ba00d2110d0, State: Running, Role: LEADER
I20260928 20:22:07.806900 16815 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1f87d7f7fc641ca9e1e8ba00d2110d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:22:07.807034 16818 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1f87d7f7fc641ca9e1e8ba00d2110d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1f87d7f7fc641ca9e1e8ba00d2110d0" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 35099 } }
I20260928 20:22:07.807422 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 16747
I20260928 20:22:07.807538 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790626764250411-3144-0/minicluster-data/master-0/wal/instance
I20260928 20:22:07.807624 16819 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1f87d7f7fc641ca9e1e8ba00d2110d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1f87d7f7fc641ca9e1e8ba00d2110d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1f87d7f7fc641ca9e1e8ba00d2110d0" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 35099 } } }
I20260928 20:22:07.807717 16819 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1f87d7f7fc641ca9e1e8ba00d2110d0 [sys.catalog]: This master's current role is: LEADER
I20260928 20:22:07.807804 3144 external_mini_cluster.cc:1428] Setting key b4bd73f6e23c6671bb344a3dd9836fbc
I20260928 20:22:07.808380 16827 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:22:07.808391 16820 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1f87d7f7fc641ca9e1e8ba00d2110d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1f87d7f7fc641ca9e1e8ba00d2110d0. Latest consensus state: current_term: 1 leader_uuid: "b1f87d7f7fc641ca9e1e8ba00d2110d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1f87d7f7fc641ca9e1e8ba00d2110d0" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 35099 } } }
I20260928 20:22:07.808903 16820 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1f87d7f7fc641ca9e1e8ba00d2110d0 [sys.catalog]: This master's current role is: LEADER
I20260928 20:22:07.809219 16827 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:22:07.810940 16827 catalog_manager.cc:1347] Generated new cluster ID: 9427a4962257449aa19a7ad007344b92
I20260928 20:22:07.811002 16827 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 20:22:07 dist-test-slave-b83v krb5kdc[16734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:22:07.818657 16837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:07.808472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60570 (local address 127.3.18.62:35099)
0928 20:22:07.809109 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:07.809118 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:07.809154 (+ 36us) server_negotiation.cc:407] Connection header received
0928 20:22:07.809288 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:07.809296 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:07.809366 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:07.809606 (+ 240us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:22:07.810360 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:07.811434 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:07.812341 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:07.812760 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:07.812890 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:07.815225 (+ 2335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:22:07.815230 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:22:07.815230 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:22:07.815248 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:22:07.816992 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:07.817403 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:07.817408 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:07.817408 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:07.817537 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:07.817754 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:07.817755 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:07.817756 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:07.818011 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:22:07.818191 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:07.818271 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 20:22:07.818422 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":62,"threads_started":1}
I20260928 20:22:07.819613 3144 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 20:22:07.821231 16827 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:22:07.821862 16827 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:22:07.826742 16827 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1f87d7f7fc641ca9e1e8ba00d2110d0: Generated new TSK 0
I20260928 20:22:07.826972 16827 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 20:22:07.873790 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 16747
2026-09-28T20:22:07Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1374 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:22:07 dist-test-slave-b83v krb5kdc[16843](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:22:07 dist-test-slave-b83v krb5kdc[16843](info): set up 2 sockets
Sep 28 20:22:07 dist-test-slave-b83v krb5kdc[16843](info): commencing operation
krb5kdc: starting...
W20260928 20:22:08.932528 3144 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Sep 28 20:22:09 dist-test-slave-b83v krb5kdc[16843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:22:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:22:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.18.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab.
Entry for principal kudu/127.3.18.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab.
I20260928 20:22:09.063634 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:43799
--webserver_interface=127.3.18.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:45035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.18.62:43799
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:22:09.146380 16856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:22:09.146622 16856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:22:09.146647 16856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:22:09.147854 16856 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:22:09.147898 16856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:22:09.147912 16856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:22:09.147924 16856 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:22:09.147936 16856 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:22:09.149479 16856 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:45035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.18.62:43799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:43799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.18.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:22:09.149693 16856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:22:09.149869 16856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:22:09.155272 16861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:22:09.155475 16862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:22:09.155475 16864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:22:09.155704 16856 server_base.cc:1034] running on GCE node
I20260928 20:22:09.156009 16856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:22:09.156327 16856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:22:09.157546 16856 hybrid_clock.cc:648] HybridClock initialized: now 1790626929157470 us; error 41 us; skew 500 ppm
Sep 28 20:22:09 dist-test-slave-b83v krb5kdc[16843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626929, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:22:09.165617 16856 init.cc:377] Logged in from keytab as kudu/127.3.18.62@KRBTEST.COM (short username kudu)
I20260928 20:22:09.166167 16856 webserver.cc:460] Webserver started at http://127.3.18.62:37001/ using document root <none> and password file <none>
I20260928 20:22:09.166378 16856 fs_manager.cc:359] Metadata directory not provided
I20260928 20:22:09.166430 16856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:22:09.166540 16856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:22:09.167630 16856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/master-0/data/instance:
uuid: "bd585a8d7a1245e89eed0a0ea71abc14"
format_stamp: "Formatted at 2026-09-28 20:22:09 on dist-test-slave-b83v"
server_key: "b67d379996b5570a2216fda6f0d9a5e6"
server_key_iv: "b81faaab8880c8a28e7c0a47e42710b3"
server_key_version: "encryptionkey@0"
I20260928 20:22:09.168076 16856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/master-0/wal/instance:
uuid: "bd585a8d7a1245e89eed0a0ea71abc14"
format_stamp: "Formatted at 2026-09-28 20:22:09 on dist-test-slave-b83v"
server_key: "b67d379996b5570a2216fda6f0d9a5e6"
server_key_iv: "b81faaab8880c8a28e7c0a47e42710b3"
server_key_version: "encryptionkey@0"
I20260928 20:22:09.169615 16856 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:22:09.170523 16871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:22:09.170873 16856 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:22:09.170964 16856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/master-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/master-0/wal
uuid: "bd585a8d7a1245e89eed0a0ea71abc14"
format_stamp: "Formatted at 2026-09-28 20:22:09 on dist-test-slave-b83v"
server_key: "b67d379996b5570a2216fda6f0d9a5e6"
server_key_iv: "b81faaab8880c8a28e7c0a47e42710b3"
server_key_version: "encryptionkey@0"
I20260928 20:22:09.171101 16856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:22:09.187000 16856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:22:09.189920 16856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:22:09.190074 16856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:22:09.195111 16856 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.62:43799
I20260928 20:22:09.195145 16923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.62:43799 every 8 connection(s)
I20260928 20:22:09.195538 16856 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
I20260928 20:22:09.196877 16924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:22:09.199159 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 16856
I20260928 20:22:09.199314 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/master-0/wal/instance
I20260928 20:22:09.199537 3144 external_mini_cluster.cc:1428] Setting key 9c571db3bc9f7d20083cd78cdaf38fcc
I20260928 20:22:09.200532 16924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd585a8d7a1245e89eed0a0ea71abc14: Bootstrap starting.
I20260928 20:22:09.201241 16924 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd585a8d7a1245e89eed0a0ea71abc14: Neither blocks nor log segments found. Creating new log.
I20260928 20:22:09.201632 16924 log.cc:824] T 00000000000000000000000000000000 P bd585a8d7a1245e89eed0a0ea71abc14: Log is configured to *not* fsync() on all Append() calls
I20260928 20:22:09.202594 16924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd585a8d7a1245e89eed0a0ea71abc14: No bootstrap required, opened a new log
I20260928 20:22:09.205281 16924 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd585a8d7a1245e89eed0a0ea71abc14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd585a8d7a1245e89eed0a0ea71abc14" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 43799 } }
I20260928 20:22:09.205476 16924 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd585a8d7a1245e89eed0a0ea71abc14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:22:09.205500 16924 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd585a8d7a1245e89eed0a0ea71abc14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd585a8d7a1245e89eed0a0ea71abc14, State: Initialized, Role: FOLLOWER
I20260928 20:22:09.205690 16924 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd585a8d7a1245e89eed0a0ea71abc14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd585a8d7a1245e89eed0a0ea71abc14" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 43799 } }
I20260928 20:22:09.205816 16924 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd585a8d7a1245e89eed0a0ea71abc14 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:22:09.205933 16924 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd585a8d7a1245e89eed0a0ea71abc14 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:22:09.206002 16924 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd585a8d7a1245e89eed0a0ea71abc14 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:22:09.206976 16924 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd585a8d7a1245e89eed0a0ea71abc14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd585a8d7a1245e89eed0a0ea71abc14" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 43799 } }
I20260928 20:22:09.207131 16924 leader_election.cc:302] T 00000000000000000000000000000000 P bd585a8d7a1245e89eed0a0ea71abc14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd585a8d7a1245e89eed0a0ea71abc14; no voters:
I20260928 20:22:09.207357 16924 leader_election.cc:288] T 00000000000000000000000000000000 P bd585a8d7a1245e89eed0a0ea71abc14 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:22:09.207446 16929 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd585a8d7a1245e89eed0a0ea71abc14 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:22:09.207695 16924 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd585a8d7a1245e89eed0a0ea71abc14 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:22:09.207983 16929 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd585a8d7a1245e89eed0a0ea71abc14 [term 1 LEADER]: Becoming Leader. State: Replica: bd585a8d7a1245e89eed0a0ea71abc14, State: Running, Role: LEADER
I20260928 20:22:09.208153 16929 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd585a8d7a1245e89eed0a0ea71abc14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd585a8d7a1245e89eed0a0ea71abc14" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 43799 } }
Sep 28 20:22:09 dist-test-slave-b83v krb5kdc[16843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:22:09.208787 16929 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd585a8d7a1245e89eed0a0ea71abc14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd585a8d7a1245e89eed0a0ea71abc14. Latest consensus state: current_term: 1 leader_uuid: "bd585a8d7a1245e89eed0a0ea71abc14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd585a8d7a1245e89eed0a0ea71abc14" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 43799 } } }
I20260928 20:22:09.208874 16929 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd585a8d7a1245e89eed0a0ea71abc14 [sys.catalog]: This master's current role is: LEADER
I20260928 20:22:09.209045 16930 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd585a8d7a1245e89eed0a0ea71abc14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd585a8d7a1245e89eed0a0ea71abc14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd585a8d7a1245e89eed0a0ea71abc14" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 43799 } } }
I20260928 20:22:09.209102 16930 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd585a8d7a1245e89eed0a0ea71abc14 [sys.catalog]: This master's current role is: LEADER
I20260928 20:22:09.209107 16938 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:22:09.210053 16938 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:22:09.213080 16938 catalog_manager.cc:1347] Generated new cluster ID: b14de213dd4c4000b5f505e94ede52e1
I20260928 20:22:09.213148 16938 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:22:09.214193 16927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:09.200239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53806 (local address 127.3.18.62:43799)
0928 20:22:09.200562 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:09.200571 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:09.200629 (+ 58us) server_negotiation.cc:407] Connection header received
0928 20:22:09.201152 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:09.201160 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:09.201217 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:09.201576 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:22:09.202457 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:09.203504 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:09.204855 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:09.205720 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:09.205910 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:09.209576 (+ 3666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:22:09.209582 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:22:09.209582 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:22:09.209598 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:22:09.212138 (+ 2540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:09.212682 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:09.212687 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:09.212687 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:09.212819 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:09.213103 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:09.213104 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:09.213105 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:09.213461 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:22:09.213627 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:09.213778 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 20:22:09.213937 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":206,"threads_started":1}
I20260928 20:22:09.224925 16938 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:22:09.225502 16938 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:22:09.229121 16938 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd585a8d7a1245e89eed0a0ea71abc14: Generated new TSK 0
I20260928 20:22:09.229327 16938 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.18.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab.
Entry for principal kudu/127.3.18.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab.
I20260928 20:22:09.263410 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.1:0
--local_ip_for_outbound_sockets=127.3.18.1
--webserver_interface=127.3.18.1
--webserver_port=0
--tserver_master_addrs=127.3.18.62:43799
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab
--principal=kudu/127.3.18.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:45035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:22:09.353791 16950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:22:09.354035 16950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:22:09.354065 16950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:22:09.355361 16950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:22:09.355407 16950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:22:09.355435 16950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.1
I20260928 20:22:09.357326 16950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:45035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab
--principal=kudu/127.3.18.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.18.1
--webserver_port=0
--tserver_master_addrs=127.3.18.62:43799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.1
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:22:09.357724 16950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:22:09.358014 16950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:22:09.363900 16956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:22:09.363901 16955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:22:09.364140 16950 server_base.cc:1034] running on GCE node
W20260928 20:22:09.363938 16958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:22:09.364423 16950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:22:09.364735 16950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:22:09.365959 16950 hybrid_clock.cc:648] HybridClock initialized: now 1790626929365938 us; error 33 us; skew 500 ppm
Sep 28 20:22:09 dist-test-slave-b83v krb5kdc[16843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626929, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:22:09.369292 16950 init.cc:377] Logged in from keytab as kudu/127.3.18.1@KRBTEST.COM (short username kudu)
I20260928 20:22:09.369871 16950 webserver.cc:460] Webserver started at http://127.3.18.1:41243/ using document root <none> and password file <none>
I20260928 20:22:09.370131 16950 fs_manager.cc:359] Metadata directory not provided
I20260928 20:22:09.370209 16950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:22:09.370354 16950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:22:09.371408 16950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-0/data/instance:
uuid: "3774a634d815476a80a6b57ac76edfd3"
format_stamp: "Formatted at 2026-09-28 20:22:09 on dist-test-slave-b83v"
server_key: "12103322db7fb85c6d347caf8fc9547b"
server_key_iv: "4ed241b44599fc4a7a075a13efb90dde"
server_key_version: "encryptionkey@0"
I20260928 20:22:09.371821 16950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-0/wal/instance:
uuid: "3774a634d815476a80a6b57ac76edfd3"
format_stamp: "Formatted at 2026-09-28 20:22:09 on dist-test-slave-b83v"
server_key: "12103322db7fb85c6d347caf8fc9547b"
server_key_iv: "4ed241b44599fc4a7a075a13efb90dde"
server_key_version: "encryptionkey@0"
I20260928 20:22:09.373207 16950 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:22:09.374017 16965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:22:09.374264 16950 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:22:09.374372 16950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-0/wal
uuid: "3774a634d815476a80a6b57ac76edfd3"
format_stamp: "Formatted at 2026-09-28 20:22:09 on dist-test-slave-b83v"
server_key: "12103322db7fb85c6d347caf8fc9547b"
server_key_iv: "4ed241b44599fc4a7a075a13efb90dde"
server_key_version: "encryptionkey@0"
I20260928 20:22:09.374472 16950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:22:09.383934 16950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:22:09.386318 16950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:22:09.386504 16950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:22:09.386754 16950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:22:09.387104 16950 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:22:09.387161 16950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:22:09.387203 16950 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:22:09.387228 16950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:22:09.393940 16950 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.1:34351
I20260928 20:22:09.393994 17078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.1:34351 every 8 connection(s)
I20260928 20:22:09.394359 16950 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
I20260928 20:22:09.398850 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 16950
I20260928 20:22:09.398969 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-0/wal/instance
I20260928 20:22:09.399267 3144 external_mini_cluster.cc:1428] Setting key 383a1908f1559276471e5685a5e37e51
Sep 28 20:22:09 dist-test-slave-b83v krb5kdc[16843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626929, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.1@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:22:09.406376 16927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:09.395250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:45825 (local address 127.3.18.62:43799)
0928 20:22:09.395319 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:09.395321 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:09.395612 (+ 291us) server_negotiation.cc:407] Connection header received
0928 20:22:09.396169 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:09.396170 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:09.396181 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:09.396241 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:22:09.397317 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:09.397906 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:09.399262 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:09.399672 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:09.399752 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:09.402881 (+ 3129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:22:09.402883 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:22:09.402883 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:22:09.402893 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:22:09.404685 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:09.405113 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:09.405115 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:09.405116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:09.405188 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:09.405533 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:09.405534 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:09.405534 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:09.405701 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:22:09.405783 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:09.406134 (+ 351us) server_negotiation.cc:299] Negotiation successful
0928 20:22:09.406243 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 20:22:09.406401 17081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:09.395236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:43799 (local address 127.3.18.1:45825)
0928 20:22:09.395425 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:09.395467 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 20:22:09.396029 (+ 562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:22:09.396362 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:22:09.396366 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:22:09.396997 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:22:09.397209 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:09.397214 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:09.398101 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:09.398102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:09.399164 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:09.399168 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:09.399829 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:09.399830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:09.400025 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:09.400748 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:22:09.400760 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:22:09.402769 (+ 2009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:22:09.404778 (+ 2009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:22:09.404785 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:22:09.404787 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:22:09.405017 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:22:09.405309 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:22:09.405310 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:22:09.405310 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:22:09.405446 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:22:09.405799 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:22:09.405806 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:22:09.405945 (+ 139us) client_negotiation.cc:769] Sending connection context
0928 20:22:09.406074 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 20:22:09.406218 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":87,"threads_started":1}
I20260928 20:22:09.407210 17079 heartbeater.cc:347] Connected to a master server at 127.3.18.62:43799
I20260928 20:22:09.407337 17079 heartbeater.cc:464] Registering TS with master...
I20260928 20:22:09.407579 17079 heartbeater.cc:511] Master 127.3.18.62:43799 requested a full tablet report, sending...
I20260928 20:22:09.408257 16888 ts_manager.cc:194] Registered new tserver with Master: 3774a634d815476a80a6b57ac76edfd3 (127.3.18.1:34351)
WARNING: no policy specified for kudu/127.3.18.2@KRBTEST.COM; defaulting to no policy
I20260928 20:22:09.409292 16888 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.18.1@KRBTEST.COM'} at 127.3.18.1:45825
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab.
Entry for principal kudu/127.3.18.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab.
I20260928 20:22:09.427064 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.2:0
--local_ip_for_outbound_sockets=127.3.18.2
--webserver_interface=127.3.18.2
--webserver_port=0
--tserver_master_addrs=127.3.18.62:43799
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab
--principal=kudu/127.3.18.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:45035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:22:09.514458 17084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:22:09.514715 17084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:22:09.514745 17084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:22:09.516160 17084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:22:09.516222 17084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:22:09.516253 17084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.2
I20260928 20:22:09.517974 17084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:45035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab
--principal=kudu/127.3.18.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.18.2
--webserver_port=0
--tserver_master_addrs=127.3.18.62:43799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.2
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:22:09.518298 17084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:22:09.518586 17084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:22:09.524901 17089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:22:09.524890 17090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:22:09.524890 17092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:22:09.525323 17084 server_base.cc:1034] running on GCE node
I20260928 20:22:09.525650 17084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:22:09.525972 17084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:22:09.527179 17084 hybrid_clock.cc:648] HybridClock initialized: now 1790626929527160 us; error 32 us; skew 500 ppm
Sep 28 20:22:09 dist-test-slave-b83v krb5kdc[16843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626929, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:22:09.530439 17084 init.cc:377] Logged in from keytab as kudu/127.3.18.2@KRBTEST.COM (short username kudu)
I20260928 20:22:09.530973 17084 webserver.cc:460] Webserver started at http://127.3.18.2:33175/ using document root <none> and password file <none>
I20260928 20:22:09.531181 17084 fs_manager.cc:359] Metadata directory not provided
I20260928 20:22:09.531229 17084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:22:09.531324 17084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:22:09.532475 17084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-1/data/instance:
uuid: "95fe79ad6abc4659bb584e646111be16"
format_stamp: "Formatted at 2026-09-28 20:22:09 on dist-test-slave-b83v"
server_key: "76b44eaf5afd3f08644d29bbfdae8e5a"
server_key_iv: "1a009335852d99adead1cf73731429dc"
server_key_version: "encryptionkey@0"
I20260928 20:22:09.532891 17084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-1/wal/instance:
uuid: "95fe79ad6abc4659bb584e646111be16"
format_stamp: "Formatted at 2026-09-28 20:22:09 on dist-test-slave-b83v"
server_key: "76b44eaf5afd3f08644d29bbfdae8e5a"
server_key_iv: "1a009335852d99adead1cf73731429dc"
server_key_version: "encryptionkey@0"
I20260928 20:22:09.534356 17084 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 20:22:09.535320 17099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:22:09.535604 17084 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:22:09.535701 17084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-1/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-1/wal
uuid: "95fe79ad6abc4659bb584e646111be16"
format_stamp: "Formatted at 2026-09-28 20:22:09 on dist-test-slave-b83v"
server_key: "76b44eaf5afd3f08644d29bbfdae8e5a"
server_key_iv: "1a009335852d99adead1cf73731429dc"
server_key_version: "encryptionkey@0"
I20260928 20:22:09.535826 17084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:22:09.551100 17084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:22:09.553913 17084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:22:09.554052 17084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:22:09.554283 17084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:22:09.554606 17084 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:22:09.554642 17084 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:22:09.554668 17084 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:22:09.554682 17084 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:22:09.561314 17084 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.2:39927
I20260928 20:22:09.561395 17212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.2:39927 every 8 connection(s)
I20260928 20:22:09.561764 17084 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
I20260928 20:22:09.562906 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 17084
I20260928 20:22:09.563030 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-1/wal/instance
I20260928 20:22:09.563234 3144 external_mini_cluster.cc:1428] Setting key 5c9e648570d715224e670391d784a470
Sep 28 20:22:09 dist-test-slave-b83v krb5kdc[16843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626929, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.2@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.18.3@KRBTEST.COM; defaulting to no policy
I20260928 20:22:09.573894 16927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:09.562755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:33833 (local address 127.3.18.62:43799)
0928 20:22:09.562847 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:09.562850 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:09.563141 (+ 291us) server_negotiation.cc:407] Connection header received
0928 20:22:09.563787 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:09.563788 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:09.563800 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:09.563855 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:22:09.565238 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:09.565964 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:09.567145 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:09.567469 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:09.567551 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:09.570464 (+ 2913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:22:09.570466 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:22:09.570466 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:22:09.570476 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:22:09.572204 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:09.572652 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:09.572654 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:09.572654 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:09.572727 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:09.573023 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:09.573025 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:09.573025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:09.573165 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:22:09.573225 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:09.573651 (+ 426us) server_negotiation.cc:299] Negotiation successful
0928 20:22:09.573782 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 20:22:09.574092 17215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:09.562719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:43799 (local address 127.3.18.2:33833)
0928 20:22:09.562935 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:09.562991 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 20:22:09.563659 (+ 668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:22:09.563919 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:22:09.563923 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:22:09.564799 (+ 876us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:22:09.565103 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:09.565110 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:09.566113 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:09.566114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:09.567033 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:09.567036 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:09.567609 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:09.567610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:09.567727 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:09.568524 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:22:09.568534 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:22:09.570354 (+ 1820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:22:09.572306 (+ 1952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:22:09.572314 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:22:09.572316 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:22:09.572564 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:22:09.572820 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:22:09.572821 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:22:09.572821 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:22:09.572949 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:22:09.573256 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:22:09.573262 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:22:09.573413 (+ 151us) client_negotiation.cc:769] Sending connection context
0928 20:22:09.573588 (+ 175us) client_negotiation.cc:240] Negotiation successful
0928 20:22:09.573741 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":104,"threads_started":1}
I20260928 20:22:09.574832 17213 heartbeater.cc:347] Connected to a master server at 127.3.18.62:43799
I20260928 20:22:09.574980 17213 heartbeater.cc:464] Registering TS with master...
I20260928 20:22:09.575255 17213 heartbeater.cc:511] Master 127.3.18.62:43799 requested a full tablet report, sending...
I20260928 20:22:09.575938 16888 ts_manager.cc:194] Registered new tserver with Master: 95fe79ad6abc4659bb584e646111be16 (127.3.18.2:39927)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.3@KRBTEST.COM" created.
I20260928 20:22:09.576526 16888 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.18.2@KRBTEST.COM'} at 127.3.18.2:33833
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab.
Entry for principal kudu/127.3.18.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab.
I20260928 20:22:09.595147 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.3:0
--local_ip_for_outbound_sockets=127.3.18.3
--webserver_interface=127.3.18.3
--webserver_port=0
--tserver_master_addrs=127.3.18.62:43799
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab
--principal=kudu/127.3.18.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:45035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:22:09.677918 17218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:22:09.678138 17218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:22:09.678164 17218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:22:09.679914 17218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:22:09.679966 17218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:22:09.679996 17218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.3
I20260928 20:22:09.681558 17218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:45035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab
--principal=kudu/127.3.18.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.18.3
--webserver_port=0
--tserver_master_addrs=127.3.18.62:43799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.3
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:22:09.681844 17218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:22:09.682121 17218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:22:09.687501 17223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:22:09.687638 17218 server_base.cc:1034] running on GCE node
W20260928 20:22:09.687633 17224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:22:09.687760 17226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:22:09.688010 17218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:22:09.688323 17218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:22:09.689498 17218 hybrid_clock.cc:648] HybridClock initialized: now 1790626929689468 us; error 51 us; skew 500 ppm
Sep 28 20:22:09 dist-test-slave-b83v krb5kdc[16843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626929, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:22:09.692600 17218 init.cc:377] Logged in from keytab as kudu/127.3.18.3@KRBTEST.COM (short username kudu)
I20260928 20:22:09.693071 17218 webserver.cc:460] Webserver started at http://127.3.18.3:33697/ using document root <none> and password file <none>
I20260928 20:22:09.693393 17218 fs_manager.cc:359] Metadata directory not provided
I20260928 20:22:09.693544 17218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:22:09.693742 17218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:22:09.694964 17218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-2/data/instance:
uuid: "f8cdfd447c5d4a09bdcc7cce530c7927"
format_stamp: "Formatted at 2026-09-28 20:22:09 on dist-test-slave-b83v"
server_key: "00013873962e4a4a6f8c35699921c4c9"
server_key_iv: "631a85baaef20a20468698ebd11638a6"
server_key_version: "encryptionkey@0"
I20260928 20:22:09.695384 17218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-2/wal/instance:
uuid: "f8cdfd447c5d4a09bdcc7cce530c7927"
format_stamp: "Formatted at 2026-09-28 20:22:09 on dist-test-slave-b83v"
server_key: "00013873962e4a4a6f8c35699921c4c9"
server_key_iv: "631a85baaef20a20468698ebd11638a6"
server_key_version: "encryptionkey@0"
I20260928 20:22:09.697099 17218 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260928 20:22:09.698159 17233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:22:09.698436 17218 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:22:09.698603 17218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-2/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-2/wal
uuid: "f8cdfd447c5d4a09bdcc7cce530c7927"
format_stamp: "Formatted at 2026-09-28 20:22:09 on dist-test-slave-b83v"
server_key: "00013873962e4a4a6f8c35699921c4c9"
server_key_iv: "631a85baaef20a20468698ebd11638a6"
server_key_version: "encryptionkey@0"
I20260928 20:22:09.698705 17218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:22:09.712323 17218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:22:09.715106 17218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:22:09.715283 17218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:22:09.715603 17218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:22:09.716097 17218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:22:09.716166 17218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:22:09.716207 17218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:22:09.716238 17218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:22:09.723088 17218 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.3:42799
I20260928 20:22:09.723191 17346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.3:42799 every 8 connection(s)
I20260928 20:22:09.723547 17218 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
Sep 28 20:22:09 dist-test-slave-b83v krb5kdc[16843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626929, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.3@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:22:09.730868 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 17218
I20260928 20:22:09.731000 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-2/wal/instance
I20260928 20:22:09.731189 3144 external_mini_cluster.cc:1428] Setting key 2a2b1259bc04606045a61f43b30beee3
I20260928 20:22:09.734313 16927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:09.724463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:36625 (local address 127.3.18.62:43799)
0928 20:22:09.724525 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:09.724527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:09.724717 (+ 190us) server_negotiation.cc:407] Connection header received
0928 20:22:09.725298 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:09.725299 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:09.725310 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:09.725374 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:22:09.726390 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:09.726904 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:09.728049 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:09.728274 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:09.728341 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:09.730869 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:22:09.730871 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:22:09.730871 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:22:09.730887 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:22:09.732756 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:09.733278 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:09.733280 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:09.733280 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:09.733347 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:09.733618 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:09.733619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:09.733619 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:09.733751 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:22:09.733827 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:09.734128 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 20:22:09.734213 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 20:22:09.734531 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:09.724453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:43799 (local address 127.3.18.3:36625)
0928 20:22:09.724607 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:09.724644 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 20:22:09.725146 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:22:09.725464 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:22:09.725467 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:22:09.726098 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:22:09.726293 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:09.726298 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:09.726987 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:09.726988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:09.727929 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:09.727931 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:09.728363 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:09.728365 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:09.728478 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:09.729077 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:22:09.729085 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:22:09.730754 (+ 1669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:22:09.732846 (+ 2092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:22:09.732855 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:22:09.732858 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:22:09.733200 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:22:09.733417 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:22:09.733418 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:22:09.733418 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:22:09.733547 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:22:09.733818 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:22:09.733825 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:22:09.733994 (+ 169us) client_negotiation.cc:769] Sending connection context
0928 20:22:09.734108 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 20:22:09.734285 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":85,"threads_started":1}
I20260928 20:22:09.735000 17347 heartbeater.cc:347] Connected to a master server at 127.3.18.62:43799
I20260928 20:22:09.735090 17347 heartbeater.cc:464] Registering TS with master...
I20260928 20:22:09.735303 17347 heartbeater.cc:511] Master 127.3.18.62:43799 requested a full tablet report, sending...
I20260928 20:22:09.735687 16888 ts_manager.cc:194] Registered new tserver with Master: f8cdfd447c5d4a09bdcc7cce530c7927 (127.3.18.3:42799)
I20260928 20:22:09.736101 16888 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.18.3@KRBTEST.COM'} at 127.3.18.3:36625
I20260928 20:22:09.743398 3144 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 20:22:09.752043 16927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:09.744899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53814 (local address 127.3.18.62:43799)
0928 20:22:09.744965 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:09.744967 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:09.745050 (+ 83us) server_negotiation.cc:407] Connection header received
0928 20:22:09.745109 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:09.745110 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:09.745119 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:09.745168 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:22:09.745838 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:09.746332 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:09.747177 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:09.747359 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:09.747430 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:09.748943 (+ 1513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:22:09.748945 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:22:09.748945 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:22:09.748952 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:22:09.750470 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:09.750935 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:09.750936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:09.750936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:09.750999 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:09.751262 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:09.751264 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:09.751264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:09.751459 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:22:09.751552 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:09.751819 (+ 267us) server_negotiation.cc:299] Negotiation successful
0928 20:22:09.751915 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 20:22:09.754122 16888 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53814:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 20:22:09.755937 16888 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 20:22:09.763861 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:09.759460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48894 (local address 127.3.18.2:39927)
0928 20:22:09.759974 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:09.759978 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:09.759988 (+ 10us) server_negotiation.cc:407] Connection header received
0928 20:22:09.760031 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:09.760033 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:09.760049 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:09.760136 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:22:09.760336 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:09.761438 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:09.762789 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:09.763296 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:09.763393 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:09.763463 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:09.763670 (+ 207us) server_negotiation.cc:299] Negotiation successful
0928 20:22:09.763719 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":77,"threads_started":1}
I20260928 20:22:09.763851 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:09.759069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.2:39927 (local address 127.0.0.1:48894)
0928 20:22:09.759642 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:09.759655 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 20:22:09.759719 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:22:09.760191 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:22:09.760192 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:22:09.760194 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:22:09.760252 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:09.760254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:09.761551 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:09.761552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:09.762700 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:09.762702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:09.763392 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:09.763393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:09.763511 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:09.763528 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 20:22:09.763649 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 20:22:09.763715 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"mutex_wait_us":46,"thread_start_us":80,"threads_started":1}
I20260928 20:22:09.763880 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:09.759401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37596 (local address 127.3.18.3:42799)
0928 20:22:09.759696 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:09.759701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:09.760107 (+ 406us) server_negotiation.cc:407] Connection header received
0928 20:22:09.760175 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:09.760177 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:09.760193 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:09.760268 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:22:09.760374 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:09.761437 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:09.762640 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:09.763361 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:09.763438 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:09.763498 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:09.763728 (+ 230us) server_negotiation.cc:299] Negotiation successful
0928 20:22:09.763768 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":103,"threads_started":1}
I20260928 20:22:09.764279 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:09.759238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56896 (local address 127.3.18.1:34351)
0928 20:22:09.759723 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:09.759729 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:09.759739 (+ 10us) server_negotiation.cc:407] Connection header received
0928 20:22:09.759810 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:09.759813 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:09.759828 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:09.759938 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:22:09.760321 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:09.761116 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:09.762729 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:09.763418 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:09.763476 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:09.763528 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:09.764136 (+ 608us) server_negotiation.cc:299] Negotiation successful
0928 20:22:09.764190 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":95,"threads_started":1}
I20260928 20:22:09.764313 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:09.759246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.3:42799 (local address 127.0.0.1:37596)
0928 20:22:09.760050 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:09.760059 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 20:22:09.760110 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:22:09.760268 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:22:09.760269 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:22:09.760272 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:22:09.760315 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:09.760316 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:09.761559 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:09.761560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:09.762537 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:09.762540 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:09.763562 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:09.763563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:09.763628 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:09.763636 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 20:22:09.764119 (+ 483us) client_negotiation.cc:240] Negotiation successful
0928 20:22:09.764169 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":785,"mutex_wait_us":33,"thread_start_us":84,"threads_started":1}
I20260928 20:22:09.763875 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:09.759027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.1:34351 (local address 127.0.0.1:56896)
0928 20:22:09.759216 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:09.759283 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0928 20:22:09.759361 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:22:09.759959 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:22:09.759962 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:22:09.759965 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:22:09.760096 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:09.760099 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:09.761225 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:09.761227 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:09.762646 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:09.762649 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:09.763570 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:09.763570 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:09.763627 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:09.763634 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 20:22:09.763685 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 20:22:09.763739 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":2,"thread_start_us":57,"threads_started":1}
I20260928 20:22:09.764956 17147 tablet_service.cc:1467] Processing CreateTablet for tablet 2d7733443bac4a7cb397a65e91007560 (DEFAULT_TABLE table=test-table [id=81284af5b52c45fda173e949de1b2f83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:22:09.765060 17281 tablet_service.cc:1467] Processing CreateTablet for tablet 2d7733443bac4a7cb397a65e91007560 (DEFAULT_TABLE table=test-table [id=81284af5b52c45fda173e949de1b2f83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:22:09.765353 17147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d7733443bac4a7cb397a65e91007560. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:22:09.765313 17013 tablet_service.cc:1467] Processing CreateTablet for tablet 2d7733443bac4a7cb397a65e91007560 (DEFAULT_TABLE table=test-table [id=81284af5b52c45fda173e949de1b2f83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:22:09.765434 17281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d7733443bac4a7cb397a65e91007560. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:22:09.765621 17013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d7733443bac4a7cb397a65e91007560. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:22:09.768280 17364 tablet_bootstrap.cc:492] T 2d7733443bac4a7cb397a65e91007560 P f8cdfd447c5d4a09bdcc7cce530c7927: Bootstrap starting.
I20260928 20:22:09.769004 17366 tablet_bootstrap.cc:492] T 2d7733443bac4a7cb397a65e91007560 P 3774a634d815476a80a6b57ac76edfd3: Bootstrap starting.
I20260928 20:22:09.769201 17364 tablet_bootstrap.cc:654] T 2d7733443bac4a7cb397a65e91007560 P f8cdfd447c5d4a09bdcc7cce530c7927: Neither blocks nor log segments found. Creating new log.
I20260928 20:22:09.769775 17366 tablet_bootstrap.cc:654] T 2d7733443bac4a7cb397a65e91007560 P 3774a634d815476a80a6b57ac76edfd3: Neither blocks nor log segments found. Creating new log.
I20260928 20:22:09.769824 17364 log.cc:824] T 2d7733443bac4a7cb397a65e91007560 P f8cdfd447c5d4a09bdcc7cce530c7927: Log is configured to *not* fsync() on all Append() calls
I20260928 20:22:09.770398 17366 log.cc:824] T 2d7733443bac4a7cb397a65e91007560 P 3774a634d815476a80a6b57ac76edfd3: Log is configured to *not* fsync() on all Append() calls
I20260928 20:22:09.770772 17364 tablet_bootstrap.cc:492] T 2d7733443bac4a7cb397a65e91007560 P f8cdfd447c5d4a09bdcc7cce530c7927: No bootstrap required, opened a new log
I20260928 20:22:09.770772 17365 tablet_bootstrap.cc:492] T 2d7733443bac4a7cb397a65e91007560 P 95fe79ad6abc4659bb584e646111be16: Bootstrap starting.
I20260928 20:22:09.770854 17364 ts_tablet_manager.cc:1397] T 2d7733443bac4a7cb397a65e91007560 P f8cdfd447c5d4a09bdcc7cce530c7927: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 20:22:09.771168 17366 tablet_bootstrap.cc:492] T 2d7733443bac4a7cb397a65e91007560 P 3774a634d815476a80a6b57ac76edfd3: No bootstrap required, opened a new log
I20260928 20:22:09.771265 17366 ts_tablet_manager.cc:1397] T 2d7733443bac4a7cb397a65e91007560 P 3774a634d815476a80a6b57ac76edfd3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 20:22:09.771406 17365 tablet_bootstrap.cc:654] T 2d7733443bac4a7cb397a65e91007560 P 95fe79ad6abc4659bb584e646111be16: Neither blocks nor log segments found. Creating new log.
I20260928 20:22:09.772037 17365 log.cc:824] T 2d7733443bac4a7cb397a65e91007560 P 95fe79ad6abc4659bb584e646111be16: Log is configured to *not* fsync() on all Append() calls
I20260928 20:22:09.772873 17365 tablet_bootstrap.cc:492] T 2d7733443bac4a7cb397a65e91007560 P 95fe79ad6abc4659bb584e646111be16: No bootstrap required, opened a new log
I20260928 20:22:09.772959 17365 ts_tablet_manager.cc:1397] T 2d7733443bac4a7cb397a65e91007560 P 95fe79ad6abc4659bb584e646111be16: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 20:22:09.773029 17364 raft_consensus.cc:348] T 2d7733443bac4a7cb397a65e91007560 P f8cdfd447c5d4a09bdcc7cce530c7927 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95fe79ad6abc4659bb584e646111be16" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 39927 } } peers { permanent_uuid: "3774a634d815476a80a6b57ac76edfd3" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34351 } } peers { permanent_uuid: "f8cdfd447c5d4a09bdcc7cce530c7927" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 42799 } }
I20260928 20:22:09.773213 17364 raft_consensus.cc:374] T 2d7733443bac4a7cb397a65e91007560 P f8cdfd447c5d4a09bdcc7cce530c7927 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:22:09.773257 17364 raft_consensus.cc:729] T 2d7733443bac4a7cb397a65e91007560 P f8cdfd447c5d4a09bdcc7cce530c7927 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8cdfd447c5d4a09bdcc7cce530c7927, State: Initialized, Role: FOLLOWER
I20260928 20:22:09.773368 17364 consensus_queue.cc:260] T 2d7733443bac4a7cb397a65e91007560 P f8cdfd447c5d4a09bdcc7cce530c7927 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95fe79ad6abc4659bb584e646111be16" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 39927 } } peers { permanent_uuid: "3774a634d815476a80a6b57ac76edfd3" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34351 } } peers { permanent_uuid: "f8cdfd447c5d4a09bdcc7cce530c7927" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 42799 } }
I20260928 20:22:09.773671 17364 ts_tablet_manager.cc:1428] T 2d7733443bac4a7cb397a65e91007560 P f8cdfd447c5d4a09bdcc7cce530c7927: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 20:22:09.773561 17366 raft_consensus.cc:348] T 2d7733443bac4a7cb397a65e91007560 P 3774a634d815476a80a6b57ac76edfd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95fe79ad6abc4659bb584e646111be16" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 39927 } } peers { permanent_uuid: "3774a634d815476a80a6b57ac76edfd3" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34351 } } peers { permanent_uuid: "f8cdfd447c5d4a09bdcc7cce530c7927" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 42799 } }
I20260928 20:22:09.773731 17366 raft_consensus.cc:374] T 2d7733443bac4a7cb397a65e91007560 P 3774a634d815476a80a6b57ac76edfd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:22:09.773787 17347 heartbeater.cc:503] Master 127.3.18.62:43799 was elected leader, sending a full tablet report...
I20260928 20:22:09.773787 17366 raft_consensus.cc:729] T 2d7733443bac4a7cb397a65e91007560 P 3774a634d815476a80a6b57ac76edfd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3774a634d815476a80a6b57ac76edfd3, State: Initialized, Role: FOLLOWER
I20260928 20:22:09.773903 17366 consensus_queue.cc:260] T 2d7733443bac4a7cb397a65e91007560 P 3774a634d815476a80a6b57ac76edfd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95fe79ad6abc4659bb584e646111be16" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 39927 } } peers { permanent_uuid: "3774a634d815476a80a6b57ac76edfd3" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34351 } } peers { permanent_uuid: "f8cdfd447c5d4a09bdcc7cce530c7927" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 42799 } }
I20260928 20:22:09.774212 17366 ts_tablet_manager.cc:1428] T 2d7733443bac4a7cb397a65e91007560 P 3774a634d815476a80a6b57ac76edfd3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 20:22:09.774235 17079 heartbeater.cc:503] Master 127.3.18.62:43799 was elected leader, sending a full tablet report...
I20260928 20:22:09.775374 17365 raft_consensus.cc:348] T 2d7733443bac4a7cb397a65e91007560 P 95fe79ad6abc4659bb584e646111be16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95fe79ad6abc4659bb584e646111be16" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 39927 } } peers { permanent_uuid: "3774a634d815476a80a6b57ac76edfd3" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34351 } } peers { permanent_uuid: "f8cdfd447c5d4a09bdcc7cce530c7927" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 42799 } }
I20260928 20:22:09.775552 17365 raft_consensus.cc:374] T 2d7733443bac4a7cb397a65e91007560 P 95fe79ad6abc4659bb584e646111be16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:22:09.775579 17365 raft_consensus.cc:729] T 2d7733443bac4a7cb397a65e91007560 P 95fe79ad6abc4659bb584e646111be16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95fe79ad6abc4659bb584e646111be16, State: Initialized, Role: FOLLOWER
I20260928 20:22:09.775676 17365 consensus_queue.cc:260] T 2d7733443bac4a7cb397a65e91007560 P 95fe79ad6abc4659bb584e646111be16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95fe79ad6abc4659bb584e646111be16" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 39927 } } peers { permanent_uuid: "3774a634d815476a80a6b57ac76edfd3" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34351 } } peers { permanent_uuid: "f8cdfd447c5d4a09bdcc7cce530c7927" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 42799 } }
I20260928 20:22:09.775987 17365 ts_tablet_manager.cc:1428] T 2d7733443bac4a7cb397a65e91007560 P 95fe79ad6abc4659bb584e646111be16: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 20:22:09.776069 17213 heartbeater.cc:503] Master 127.3.18.62:43799 was elected leader, sending a full tablet report...
I20260928 20:22:09.802799 17371 raft_consensus.cc:482] T 2d7733443bac4a7cb397a65e91007560 P 3774a634d815476a80a6b57ac76edfd3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:22:09.802950 17371 raft_consensus.cc:504] T 2d7733443bac4a7cb397a65e91007560 P 3774a634d815476a80a6b57ac76edfd3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95fe79ad6abc4659bb584e646111be16" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 39927 } } peers { permanent_uuid: "3774a634d815476a80a6b57ac76edfd3" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34351 } } peers { permanent_uuid: "f8cdfd447c5d4a09bdcc7cce530c7927" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 42799 } }
I20260928 20:22:09.803304 17371 leader_election.cc:288] T 2d7733443bac4a7cb397a65e91007560 P 3774a634d815476a80a6b57ac76edfd3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 95fe79ad6abc4659bb584e646111be16 (127.3.18.2:39927), f8cdfd447c5d4a09bdcc7cce530c7927 (127.3.18.3:42799)
I20260928 20:22:09.806370 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:09.803569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:33303 (local address 127.3.18.3:42799)
0928 20:22:09.803630 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:09.803632 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:09.803641 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:22:09.803670 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:09.803671 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:09.803683 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:09.803744 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:22:09.804001 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:09.804557 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:09.805552 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:09.805980 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:09.806031 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:09.806040 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:09.806254 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 20:22:09.806296 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 20:22:09.806411 17081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:09.803473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.3:42799 (local address 127.3.18.1:33303)
0928 20:22:09.803525 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:09.803537 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:22:09.803585 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:22:09.803804 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:22:09.803806 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:22:09.803808 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:22:09.803892 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:09.803895 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:09.804663 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:09.804664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:09.805455 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:09.805458 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:09.806065 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:09.806066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:09.806145 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:09.806150 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 20:22:09.806228 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 20:22:09.806277 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 20:22:09.806620 17373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:09.803481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.2:39927 (local address 127.3.18.1:36117)
0928 20:22:09.803699 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:09.803707 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 20:22:09.803778 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:22:09.803993 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:22:09.803994 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:22:09.803995 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:22:09.804066 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:09.804067 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:09.804808 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:09.804809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:09.805622 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:09.805625 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:09.806277 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:09.806278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:09.806343 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:09.806346 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 20:22:09.806405 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 20:22:09.806527 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":3,"thread_start_us":121,"threads_started":1}
I20260928 20:22:09.806766 17301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7733443bac4a7cb397a65e91007560" candidate_uuid: "3774a634d815476a80a6b57ac76edfd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8cdfd447c5d4a09bdcc7cce530c7927" is_pre_election: true
I20260928 20:22:09.806903 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:09.803621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:36117 (local address 127.3.18.2:39927)
0928 20:22:09.803668 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:09.803671 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:09.803772 (+ 101us) server_negotiation.cc:407] Connection header received
0928 20:22:09.803846 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:09.803847 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:09.803861 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:09.803930 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:22:09.804168 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:09.804714 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:09.805713 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:09.806190 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:09.806260 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:09.806269 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:09.806775 (+ 506us) server_negotiation.cc:299] Negotiation successful
0928 20:22:09.806823 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 20:22:09.806927 17301 raft_consensus.cc:2454] T 2d7733443bac4a7cb397a65e91007560 P f8cdfd447c5d4a09bdcc7cce530c7927 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3774a634d815476a80a6b57ac76edfd3 in term 0.
I20260928 20:22:09.807202 17167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7733443bac4a7cb397a65e91007560" candidate_uuid: "3774a634d815476a80a6b57ac76edfd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95fe79ad6abc4659bb584e646111be16" is_pre_election: true
I20260928 20:22:09.807288 16968 leader_election.cc:302] T 2d7733443bac4a7cb397a65e91007560 P 3774a634d815476a80a6b57ac76edfd3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3774a634d815476a80a6b57ac76edfd3, f8cdfd447c5d4a09bdcc7cce530c7927; no voters:
I20260928 20:22:09.807361 17167 raft_consensus.cc:2454] T 2d7733443bac4a7cb397a65e91007560 P 95fe79ad6abc4659bb584e646111be16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3774a634d815476a80a6b57ac76edfd3 in term 0.
I20260928 20:22:09.807425 17371 raft_consensus.cc:2781] T 2d7733443bac4a7cb397a65e91007560 P 3774a634d815476a80a6b57ac76edfd3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 20:22:09.807497 17371 raft_consensus.cc:482] T 2d7733443bac4a7cb397a65e91007560 P 3774a634d815476a80a6b57ac76edfd3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 20:22:09.807539 17371 raft_consensus.cc:3037] T 2d7733443bac4a7cb397a65e91007560 P 3774a634d815476a80a6b57ac76edfd3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:22:09.808638 17371 raft_consensus.cc:504] T 2d7733443bac4a7cb397a65e91007560 P 3774a634d815476a80a6b57ac76edfd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95fe79ad6abc4659bb584e646111be16" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 39927 } } peers { permanent_uuid: "3774a634d815476a80a6b57ac76edfd3" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34351 } } peers { permanent_uuid: "f8cdfd447c5d4a09bdcc7cce530c7927" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 42799 } }
I20260928 20:22:09.808796 17371 leader_election.cc:288] T 2d7733443bac4a7cb397a65e91007560 P 3774a634d815476a80a6b57ac76edfd3 [CANDIDATE]: Term 1 election: Requested vote from peers 95fe79ad6abc4659bb584e646111be16 (127.3.18.2:39927), f8cdfd447c5d4a09bdcc7cce530c7927 (127.3.18.3:42799)
I20260928 20:22:09.809016 17167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7733443bac4a7cb397a65e91007560" candidate_uuid: "3774a634d815476a80a6b57ac76edfd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95fe79ad6abc4659bb584e646111be16"
I20260928 20:22:09.809016 17301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7733443bac4a7cb397a65e91007560" candidate_uuid: "3774a634d815476a80a6b57ac76edfd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8cdfd447c5d4a09bdcc7cce530c7927"
I20260928 20:22:09.809149 17301 raft_consensus.cc:3037] T 2d7733443bac4a7cb397a65e91007560 P f8cdfd447c5d4a09bdcc7cce530c7927 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:22:09.809132 17167 raft_consensus.cc:3037] T 2d7733443bac4a7cb397a65e91007560 P 95fe79ad6abc4659bb584e646111be16 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:22:09.810124 17167 raft_consensus.cc:2454] T 2d7733443bac4a7cb397a65e91007560 P 95fe79ad6abc4659bb584e646111be16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3774a634d815476a80a6b57ac76edfd3 in term 1.
I20260928 20:22:09.810139 17301 raft_consensus.cc:2454] T 2d7733443bac4a7cb397a65e91007560 P f8cdfd447c5d4a09bdcc7cce530c7927 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3774a634d815476a80a6b57ac76edfd3 in term 1.
I20260928 20:22:09.810427 16966 leader_election.cc:302] T 2d7733443bac4a7cb397a65e91007560 P 3774a634d815476a80a6b57ac76edfd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3774a634d815476a80a6b57ac76edfd3, 95fe79ad6abc4659bb584e646111be16; no voters:
I20260928 20:22:09.810592 17371 raft_consensus.cc:2781] T 2d7733443bac4a7cb397a65e91007560 P 3774a634d815476a80a6b57ac76edfd3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:22:09.810752 17371 raft_consensus.cc:686] T 2d7733443bac4a7cb397a65e91007560 P 3774a634d815476a80a6b57ac76edfd3 [term 1 LEADER]: Becoming Leader. State: Replica: 3774a634d815476a80a6b57ac76edfd3, State: Running, Role: LEADER
I20260928 20:22:09.810830 17371 consensus_queue.cc:237] T 2d7733443bac4a7cb397a65e91007560 P 3774a634d815476a80a6b57ac76edfd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95fe79ad6abc4659bb584e646111be16" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 39927 } } peers { permanent_uuid: "3774a634d815476a80a6b57ac76edfd3" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34351 } } peers { permanent_uuid: "f8cdfd447c5d4a09bdcc7cce530c7927" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 42799 } }
I20260928 20:22:09.811878 16888 catalog_manager.cc:5495] T 2d7733443bac4a7cb397a65e91007560 P 3774a634d815476a80a6b57ac76edfd3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3774a634d815476a80a6b57ac76edfd3 (127.3.18.1). New cstate: current_term: 1 leader_uuid: "3774a634d815476a80a6b57ac76edfd3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95fe79ad6abc4659bb584e646111be16" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 39927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3774a634d815476a80a6b57ac76edfd3" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8cdfd447c5d4a09bdcc7cce530c7927" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 42799 } health_report { overall_health: UNKNOWN } } }
W20260928 20:22:09.812340 17214 tablet.cc:2369] T 2d7733443bac4a7cb397a65e91007560 P 95fe79ad6abc4659bb584e646111be16: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 20:22:09.823549 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:09.821059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56906 (local address 127.3.18.1:34351)
0928 20:22:09.821130 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:09.821133 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:09.821142 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:22:09.821183 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:09.821183 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:09.821203 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:09.821287 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:22:09.821446 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:09.821948 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:09.822694 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:09.822889 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:09.822984 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:09.823178 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:22:09.823288 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:22:09.823386 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:09.823404 (+ 18us) server_negotiation.cc:299] Negotiation successful
0928 20:22:09.823424 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 20:22:09.825464 17167 raft_consensus.cc:1261] T 2d7733443bac4a7cb397a65e91007560 P 95fe79ad6abc4659bb584e646111be16 [term 1 FOLLOWER]: Refusing update from remote peer 3774a634d815476a80a6b57ac76edfd3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:22:09.825456 17301 raft_consensus.cc:1261] T 2d7733443bac4a7cb397a65e91007560 P f8cdfd447c5d4a09bdcc7cce530c7927 [term 1 FOLLOWER]: Refusing update from remote peer 3774a634d815476a80a6b57ac76edfd3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:22:09.825848 17374 consensus_queue.cc:1035] T 2d7733443bac4a7cb397a65e91007560 P 3774a634d815476a80a6b57ac76edfd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8cdfd447c5d4a09bdcc7cce530c7927" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 42799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:22:09.826021 17371 consensus_queue.cc:1035] T 2d7733443bac4a7cb397a65e91007560 P 3774a634d815476a80a6b57ac76edfd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95fe79ad6abc4659bb584e646111be16" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 39927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:22:09.828930 17378 mvcc.cc:205] Tried to move back new op lower bound from 7334407904561958912 to 7334407904506028032. Current Snapshot: MvccSnapshot[applied={T|T < 7334407904561958912 or (T in {7334407904561958912})}]
I20260928 20:22:09.829622 17379 mvcc.cc:205] Tried to move back new op lower bound from 7334407904561958912 to 7334407904506028032. Current Snapshot: MvccSnapshot[applied={T|T < 7334407904561958912}]
I20260928 20:22:09.831852 16887 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:22:09.831974 16887 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:22:09.832732 16887 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bd585a8d7a1245e89eed0a0ea71abc14: Sending DeleteTablet for 3 replicas of tablet 2d7733443bac4a7cb397a65e91007560
I20260928 20:22:09.833099 17147 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d7733443bac4a7cb397a65e91007560 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:22:09 UTC) from {username='kudu', principal='kudu/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:48894
I20260928 20:22:09.833115 17013 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d7733443bac4a7cb397a65e91007560 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:22:09 UTC) from {username='kudu', principal='kudu/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:56896
I20260928 20:22:09.833148 17281 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d7733443bac4a7cb397a65e91007560 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:22:09 UTC) from {username='kudu', principal='kudu/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:37596
I20260928 20:22:09.833511 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 16950
I20260928 20:22:09.833580 17389 tablet_replica.cc:330] T 2d7733443bac4a7cb397a65e91007560 P f8cdfd447c5d4a09bdcc7cce530c7927: stopping tablet replica
I20260928 20:22:09.833688 17389 raft_consensus.cc:2229] T 2d7733443bac4a7cb397a65e91007560 P f8cdfd447c5d4a09bdcc7cce530c7927 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:22:09.833772 17389 raft_consensus.cc:2258] T 2d7733443bac4a7cb397a65e91007560 P f8cdfd447c5d4a09bdcc7cce530c7927 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:22:09.834718 17389 ts_tablet_manager.cc:1905] T 2d7733443bac4a7cb397a65e91007560 P f8cdfd447c5d4a09bdcc7cce530c7927: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:22:09.836865 17389 ts_tablet_manager.cc:1918] T 2d7733443bac4a7cb397a65e91007560 P f8cdfd447c5d4a09bdcc7cce530c7927: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:22:09.836947 17389 log.cc:1196] T 2d7733443bac4a7cb397a65e91007560 P f8cdfd447c5d4a09bdcc7cce530c7927: Deleting WAL directory at /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-2/wal/wals/2d7733443bac4a7cb397a65e91007560
I20260928 20:22:09.837230 17389 ts_tablet_manager.cc:1939] T 2d7733443bac4a7cb397a65e91007560 P f8cdfd447c5d4a09bdcc7cce530c7927: Deleting consensus metadata
I20260928 20:22:09.838196 16874 catalog_manager.cc:4841] TS f8cdfd447c5d4a09bdcc7cce530c7927 (127.3.18.3:42799): tablet 2d7733443bac4a7cb397a65e91007560 (table test-table [id=81284af5b52c45fda173e949de1b2f83]) successfully deleted
I20260928 20:22:09.839656 17387 tablet_replica.cc:330] T 2d7733443bac4a7cb397a65e91007560 P 95fe79ad6abc4659bb584e646111be16: stopping tablet replica
I20260928 20:22:09.839844 17387 raft_consensus.cc:2229] T 2d7733443bac4a7cb397a65e91007560 P 95fe79ad6abc4659bb584e646111be16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:22:09.839980 17387 raft_consensus.cc:2258] T 2d7733443bac4a7cb397a65e91007560 P 95fe79ad6abc4659bb584e646111be16 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 20:22:09.840437 16874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.18.1:34351 (error 108)
I20260928 20:22:09.840513 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 17084
I20260928 20:22:09.841856 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:09.841720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.1:34351 (local address 127.0.0.1:56908)
0928 20:22:09.841775 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:09.841815 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.1:34351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 20:22:09.842046 16874 catalog_manager.cc:4584] TS 3774a634d815476a80a6b57ac76edfd3 (127.3.18.1:34351): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2d7733443bac4a7cb397a65e91007560: Network error: Client connection negotiation failed: client connection to 127.3.18.1:34351: connect: Connection refused (error 111)
I20260928 20:22:09.846083 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 17218
I20260928 20:22:09.846204 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:09.846040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.2:39927 (local address 127.0.0.1:48908)
0928 20:22:09.846108 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:09.846145 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.2:39927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 20:22:09.850971 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 16856
I20260928 20:22:09.856153 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:43799
--webserver_interface=127.3.18.62
--webserver_port=37001
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:45035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.18.62:43799
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:22:09.939993 17391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:22:09.940243 17391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:22:09.940270 17391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:22:09.941653 17391 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:22:09.941715 17391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:22:09.941730 17391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:22:09.941742 17391 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:22:09.941756 17391 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:22:09.944103 17391 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:45035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.3.18.62:43799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:43799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.18.62
--webserver_port=37001
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17391
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:22:09.944372 17391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:22:09.944579 17391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:22:09.956727 17397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:22:09.956727 17400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:22:09.956779 17398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:22:09.957064 17391 server_base.cc:1034] running on GCE node
I20260928 20:22:09.957358 17391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:22:09.957667 17391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:22:09.958813 17391 hybrid_clock.cc:648] HybridClock initialized: now 1790626929958821 us; error 55 us; skew 500 ppm
Sep 28 20:22:09 dist-test-slave-b83v krb5kdc[16843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626929, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:22:09.961652 17391 init.cc:377] Logged in from keytab as kudu/127.3.18.62@KRBTEST.COM (short username kudu)
I20260928 20:22:09.962162 17391 webserver.cc:460] Webserver started at http://127.3.18.62:37001/ using document root <none> and password file <none>
I20260928 20:22:09.962414 17391 fs_manager.cc:359] Metadata directory not provided
I20260928 20:22:09.962508 17391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:22:09.964064 17391 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:22:09.964805 17407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:22:09.964985 17391 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:22:09.965099 17391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/master-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/master-0/wal
uuid: "bd585a8d7a1245e89eed0a0ea71abc14"
format_stamp: "Formatted at 2026-09-28 20:22:09 on dist-test-slave-b83v"
server_key: "b67d379996b5570a2216fda6f0d9a5e6"
server_key_iv: "b81faaab8880c8a28e7c0a47e42710b3"
server_key_version: "encryptionkey@0"
I20260928 20:22:09.965528 17391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:22:09.978345 17391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:22:09.981305 17391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:22:09.981510 17391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:22:09.986416 17391 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.62:43799
I20260928 20:22:09.986536 17459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.62:43799 every 8 connection(s)
I20260928 20:22:09.986917 17391 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
I20260928 20:22:09.988814 17460 sys_catalog.cc:263] Verifying existing consensus state
I20260928 20:22:09.989544 17460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd585a8d7a1245e89eed0a0ea71abc14: Bootstrap starting.
I20260928 20:22:09.991593 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 17391
I20260928 20:22:09.991987 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.1:34351
--local_ip_for_outbound_sockets=127.3.18.1
--tserver_master_addrs=127.3.18.62:43799
--webserver_port=41243
--webserver_interface=127.3.18.1
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab
--principal=kudu/127.3.18.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:45035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 20:22:09.999281 17460 log.cc:824] T 00000000000000000000000000000000 P bd585a8d7a1245e89eed0a0ea71abc14: Log is configured to *not* fsync() on all Append() calls
I20260928 20:22:10.001588 17460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd585a8d7a1245e89eed0a0ea71abc14: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 20:22:10.001929 17460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd585a8d7a1245e89eed0a0ea71abc14: Bootstrap complete.
I20260928 20:22:10.004585 17460 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd585a8d7a1245e89eed0a0ea71abc14 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd585a8d7a1245e89eed0a0ea71abc14" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 43799 } }
I20260928 20:22:10.004840 17460 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd585a8d7a1245e89eed0a0ea71abc14 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd585a8d7a1245e89eed0a0ea71abc14, State: Initialized, Role: FOLLOWER
I20260928 20:22:10.005049 17460 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd585a8d7a1245e89eed0a0ea71abc14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd585a8d7a1245e89eed0a0ea71abc14" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 43799 } }
I20260928 20:22:10.005142 17460 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd585a8d7a1245e89eed0a0ea71abc14 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:22:10.005198 17460 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd585a8d7a1245e89eed0a0ea71abc14 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:22:10.005260 17460 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd585a8d7a1245e89eed0a0ea71abc14 [term 1 FOLLOWER]: Advancing to term 2
I20260928 20:22:10.006506 17460 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd585a8d7a1245e89eed0a0ea71abc14 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd585a8d7a1245e89eed0a0ea71abc14" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 43799 } }
I20260928 20:22:10.006604 17460 leader_election.cc:302] T 00000000000000000000000000000000 P bd585a8d7a1245e89eed0a0ea71abc14 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd585a8d7a1245e89eed0a0ea71abc14; no voters:
I20260928 20:22:10.006837 17460 leader_election.cc:288] T 00000000000000000000000000000000 P bd585a8d7a1245e89eed0a0ea71abc14 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 20:22:10.006891 17464 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd585a8d7a1245e89eed0a0ea71abc14 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 20:22:10.007043 17464 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd585a8d7a1245e89eed0a0ea71abc14 [term 2 LEADER]: Becoming Leader. State: Replica: bd585a8d7a1245e89eed0a0ea71abc14, State: Running, Role: LEADER
I20260928 20:22:10.007174 17460 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd585a8d7a1245e89eed0a0ea71abc14 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:22:10.007148 17464 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd585a8d7a1245e89eed0a0ea71abc14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd585a8d7a1245e89eed0a0ea71abc14" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 43799 } }
I20260928 20:22:10.007479 17464 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd585a8d7a1245e89eed0a0ea71abc14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd585a8d7a1245e89eed0a0ea71abc14. Latest consensus state: current_term: 2 leader_uuid: "bd585a8d7a1245e89eed0a0ea71abc14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd585a8d7a1245e89eed0a0ea71abc14" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 43799 } } }
I20260928 20:22:10.007576 17464 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd585a8d7a1245e89eed0a0ea71abc14 [sys.catalog]: This master's current role is: LEADER
I20260928 20:22:10.007480 17465 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd585a8d7a1245e89eed0a0ea71abc14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "bd585a8d7a1245e89eed0a0ea71abc14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd585a8d7a1245e89eed0a0ea71abc14" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 43799 } } }
I20260928 20:22:10.007745 17465 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd585a8d7a1245e89eed0a0ea71abc14 [sys.catalog]: This master's current role is: LEADER
I20260928 20:22:10.007835 17469 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:22:10.008833 17469 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:22:10.008981 17469 catalog_manager.cc:1259] Loaded cluster ID: b14de213dd4c4000b5f505e94ede52e1
I20260928 20:22:10.009253 17469 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:22:10.009547 17469 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 20:22:10.014786 17469 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 20:22:10.017810 17469 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:22:10.018129 17469 catalog_manager.cc:5879] T 00000000000000000000000000000000 P bd585a8d7a1245e89eed0a0ea71abc14: Loaded TSK: 0
I20260928 20:22:10.018388 17469 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 20:22:10.078588 17462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:22:10.078890 17462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:22:10.078933 17462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:22:10.080371 17462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:22:10.080457 17462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:22:10.080511 17462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.1
I20260928 20:22:10.082191 17462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:45035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab
--principal=kudu/127.3.18.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.1:34351
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.18.1
--webserver_port=41243
--tserver_master_addrs=127.3.18.62:43799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.1
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:22:10.082500 17462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:22:10.082765 17462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:22:10.088336 17490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:22:10.088326 17488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:22:10.088354 17487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:22:10.088722 17462 server_base.cc:1034] running on GCE node
I20260928 20:22:10.088966 17462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:22:10.089246 17462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:22:10.090404 17462 hybrid_clock.cc:648] HybridClock initialized: now 1790626930090389 us; error 34 us; skew 500 ppm
Sep 28 20:22:10 dist-test-slave-b83v krb5kdc[16843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626930, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:22:10.093246 17462 init.cc:377] Logged in from keytab as kudu/127.3.18.1@KRBTEST.COM (short username kudu)
I20260928 20:22:10.093776 17462 webserver.cc:460] Webserver started at http://127.3.18.1:41243/ using document root <none> and password file <none>
I20260928 20:22:10.094041 17462 fs_manager.cc:359] Metadata directory not provided
I20260928 20:22:10.094105 17462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:22:10.095618 17462 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:22:10.096571 17497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:22:10.096798 17462 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:22:10.096901 17462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-0/wal
uuid: "3774a634d815476a80a6b57ac76edfd3"
format_stamp: "Formatted at 2026-09-28 20:22:09 on dist-test-slave-b83v"
server_key: "12103322db7fb85c6d347caf8fc9547b"
server_key_iv: "4ed241b44599fc4a7a075a13efb90dde"
server_key_version: "encryptionkey@0"
I20260928 20:22:10.097314 17462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:22:10.113523 17462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:22:10.116319 17462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:22:10.116483 17462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:22:10.116822 17462 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:22:10.117395 17504 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 20:22:10.119508 17462 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 20:22:10.119570 17462 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 20:22:10.119612 17462 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 20:22:10.120661 17462 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 20:22:10.120720 17462 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 20:22:10.120777 17504 tablet_bootstrap.cc:492] T 2d7733443bac4a7cb397a65e91007560 P 3774a634d815476a80a6b57ac76edfd3: Bootstrap starting.
I20260928 20:22:10.128264 17462 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.1:34351
I20260928 20:22:10.128412 17611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.1:34351 every 8 connection(s)
I20260928 20:22:10.128719 17462 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
I20260928 20:22:10.132009 17504 log.cc:824] T 2d7733443bac4a7cb397a65e91007560 P 3774a634d815476a80a6b57ac76edfd3: Log is configured to *not* fsync() on all Append() calls
I20260928 20:22:10.135556 17504 tablet_bootstrap.cc:492] T 2d7733443bac4a7cb397a65e91007560 P 3774a634d815476a80a6b57ac76edfd3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 20:22:10.135952 17504 tablet_bootstrap.cc:492] T 2d7733443bac4a7cb397a65e91007560 P 3774a634d815476a80a6b57ac76edfd3: Bootstrap complete.
Sep 28 20:22:10 dist-test-slave-b83v krb5kdc[16843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626930, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.1@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:22:10.137177 17504 ts_tablet_manager.cc:1397] T 2d7733443bac4a7cb397a65e91007560 P 3774a634d815476a80a6b57ac76edfd3: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.006s
I20260928 20:22:10.138094 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 17462
I20260928 20:22:10.138384 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.2:39927
--local_ip_for_outbound_sockets=127.3.18.2
--tserver_master_addrs=127.3.18.62:43799
--webserver_port=33175
--webserver_interface=127.3.18.2
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab
--principal=kudu/127.3.18.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:45035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 20:22:10.138649 17504 raft_consensus.cc:348] T 2d7733443bac4a7cb397a65e91007560 P 3774a634d815476a80a6b57ac76edfd3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95fe79ad6abc4659bb584e646111be16" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 39927 } } peers { permanent_uuid: "3774a634d815476a80a6b57ac76edfd3" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34351 } } peers { permanent_uuid: "f8cdfd447c5d4a09bdcc7cce530c7927" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 42799 } }
I20260928 20:22:10.139007 17504 raft_consensus.cc:729] T 2d7733443bac4a7cb397a65e91007560 P 3774a634d815476a80a6b57ac76edfd3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3774a634d815476a80a6b57ac76edfd3, State: Initialized, Role: FOLLOWER
I20260928 20:22:10.139163 17504 consensus_queue.cc:260] T 2d7733443bac4a7cb397a65e91007560 P 3774a634d815476a80a6b57ac76edfd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95fe79ad6abc4659bb584e646111be16" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 39927 } } peers { permanent_uuid: "3774a634d815476a80a6b57ac76edfd3" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34351 } } peers { permanent_uuid: "f8cdfd447c5d4a09bdcc7cce530c7927" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 42799 } }
I20260928 20:22:10.139544 17504 ts_tablet_manager.cc:1428] T 2d7733443bac4a7cb397a65e91007560 P 3774a634d815476a80a6b57ac76edfd3: Time spent starting tablet: real 0.002s user 0.000s sys 0.004s
I20260928 20:22:10.144702 17615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:10.129613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:41001 (local address 127.3.18.62:43799)
0928 20:22:10.129810 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:10.129827 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:10.130065 (+ 238us) server_negotiation.cc:407] Connection header received
0928 20:22:10.130754 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:10.130759 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:10.130805 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:10.130948 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:22:10.131596 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:10.132898 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:10.134030 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:10.134539 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:10.134674 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:10.137314 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:22:10.137320 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:22:10.137321 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:22:10.137336 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:22:10.141826 (+ 4490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:10.142546 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:10.142551 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:10.142552 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:10.142705 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:10.143109 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:10.143110 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:10.143110 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:10.143561 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:22:10.143713 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:10.144252 (+ 539us) server_negotiation.cc:299] Negotiation successful
0928 20:22:10.144483 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":87,"threads_started":1}
I20260928 20:22:10.144754 17614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:10.129512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:43799 (local address 127.3.18.1:41001)
0928 20:22:10.129901 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:10.129935 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 20:22:10.130483 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:22:10.130974 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:22:10.130978 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:22:10.131267 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:22:10.131460 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:10.131465 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:10.133022 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:10.133023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:10.133927 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:10.133929 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:10.134657 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:10.134658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:10.134798 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:10.135427 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:22:10.135436 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:22:10.137181 (+ 1745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:22:10.141982 (+ 4801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:22:10.141995 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:22:10.141997 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:22:10.142418 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:22:10.142808 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:22:10.142809 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:22:10.142810 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:22:10.143007 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:22:10.143700 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:22:10.143711 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:22:10.144006 (+ 295us) client_negotiation.cc:769] Sending connection context
0928 20:22:10.144159 (+ 153us) client_negotiation.cc:240] Negotiation successful
0928 20:22:10.144433 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":95,"threads_started":1}
I20260928 20:22:10.145490 17612 heartbeater.cc:347] Connected to a master server at 127.3.18.62:43799
I20260928 20:22:10.145596 17612 heartbeater.cc:464] Registering TS with master...
I20260928 20:22:10.145794 17612 heartbeater.cc:511] Master 127.3.18.62:43799 requested a full tablet report, sending...
I20260928 20:22:10.146777 17424 ts_manager.cc:194] Registered new tserver with Master: 3774a634d815476a80a6b57ac76edfd3 (127.3.18.1:34351)
I20260928 20:22:10.148291 17424 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.18.1@KRBTEST.COM'} at 127.3.18.1:41001
I20260928 20:22:10.149446 17612 heartbeater.cc:503] Master 127.3.18.62:43799 was elected leader, sending a full tablet report...
I20260928 20:22:10.157080 17619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:10.147600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.1:34351 (local address 127.0.0.1:56914)
0928 20:22:10.147980 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:10.148020 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 20:22:10.148090 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:22:10.152244 (+ 4154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:22:10.152249 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:22:10.152254 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:22:10.152427 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:10.152455 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:10.154035 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:10.154038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:10.155526 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:10.155532 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:10.156659 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:10.156660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:10.156788 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:10.156806 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 20:22:10.156879 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 20:22:10.156964 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":128,"threads_started":1}
I20260928 20:22:10.157764 17620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:10.148674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56914 (local address 127.3.18.1:34351)
0928 20:22:10.151954 (+ 3280us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:10.151961 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:10.151979 (+ 18us) server_negotiation.cc:407] Connection header received
0928 20:22:10.152038 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:10.152042 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:10.152063 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:10.152204 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:22:10.152583 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:10.153877 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:10.155668 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:10.156537 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:10.157504 (+ 967us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:10.157605 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:10.157637 (+ 32us) server_negotiation.cc:299] Negotiation successful
0928 20:22:10.157689 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3225,"thread_start_us":116,"threads_started":1}
I20260928 20:22:10.158263 17546 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d7733443bac4a7cb397a65e91007560 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:22:09 UTC) from {username='kudu', principal='kudu/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:56914
I20260928 20:22:10.158953 17621 tablet_replica.cc:330] stopping tablet replica
I20260928 20:22:10.159096 17621 raft_consensus.cc:2229] T 2d7733443bac4a7cb397a65e91007560 P 3774a634d815476a80a6b57ac76edfd3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:22:10.159169 17621 raft_consensus.cc:2258] T 2d7733443bac4a7cb397a65e91007560 P 3774a634d815476a80a6b57ac76edfd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:22:10.159539 17621 ts_tablet_manager.cc:1905] T 2d7733443bac4a7cb397a65e91007560 P 3774a634d815476a80a6b57ac76edfd3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:22:10.162266 17621 ts_tablet_manager.cc:1918] T 2d7733443bac4a7cb397a65e91007560 P 3774a634d815476a80a6b57ac76edfd3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:22:10.162357 17621 log.cc:1196] T 2d7733443bac4a7cb397a65e91007560 P 3774a634d815476a80a6b57ac76edfd3: Deleting WAL directory at /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-0/wal/wals/2d7733443bac4a7cb397a65e91007560
I20260928 20:22:10.162931 17621 ts_tablet_manager.cc:1939] T 2d7733443bac4a7cb397a65e91007560 P 3774a634d815476a80a6b57ac76edfd3: Deleting consensus metadata
I20260928 20:22:10.163655 17410 catalog_manager.cc:4841] TS 3774a634d815476a80a6b57ac76edfd3 (127.3.18.1:34351): tablet 2d7733443bac4a7cb397a65e91007560 (table test-table [id=81284af5b52c45fda173e949de1b2f83]) successfully deleted
W20260928 20:22:10.238409 17618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:22:10.238686 17618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:22:10.238741 17618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:22:10.240038 17618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:22:10.240111 17618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:22:10.240167 17618 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.2
I20260928 20:22:10.241777 17618 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:45035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab
--principal=kudu/127.3.18.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.2:39927
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.18.2
--webserver_port=33175
--tserver_master_addrs=127.3.18.62:43799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.2
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:22:10.242051 17618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:22:10.242288 17618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 20:22:10.247870 17618 server_base.cc:1034] running on GCE node
W20260928 20:22:10.247792 17626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:22:10.247797 17629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:22:10.247828 17627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:22:10.248334 17618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:22:10.248620 17618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:22:10.249789 17618 hybrid_clock.cc:648] HybridClock initialized: now 1790626930249774 us; error 31 us; skew 500 ppm
Sep 28 20:22:10 dist-test-slave-b83v krb5kdc[16843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626930, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:22:10.252640 17618 init.cc:377] Logged in from keytab as kudu/127.3.18.2@KRBTEST.COM (short username kudu)
I20260928 20:22:10.253111 17618 webserver.cc:460] Webserver started at http://127.3.18.2:33175/ using document root <none> and password file <none>
I20260928 20:22:10.253340 17618 fs_manager.cc:359] Metadata directory not provided
I20260928 20:22:10.253399 17618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:22:10.254877 17618 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:22:10.255646 17636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:22:10.255904 17618 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:22:10.255981 17618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-1/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-1/wal
uuid: "95fe79ad6abc4659bb584e646111be16"
format_stamp: "Formatted at 2026-09-28 20:22:09 on dist-test-slave-b83v"
server_key: "76b44eaf5afd3f08644d29bbfdae8e5a"
server_key_iv: "1a009335852d99adead1cf73731429dc"
server_key_version: "encryptionkey@0"
I20260928 20:22:10.256364 17618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:22:10.267000 17618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:22:10.269807 17618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:22:10.269958 17618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:22:10.270253 17618 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:22:10.270751 17643 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 20:22:10.272434 17618 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 20:22:10.272496 17618 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 20:22:10.272579 17618 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 20:22:10.273298 17618 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 20:22:10.273353 17618 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 20:22:10.273458 17643 tablet_bootstrap.cc:492] T 2d7733443bac4a7cb397a65e91007560 P 95fe79ad6abc4659bb584e646111be16: Bootstrap starting.
I20260928 20:22:10.279914 17618 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.2:39927
I20260928 20:22:10.280028 17750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.2:39927 every 8 connection(s)
I20260928 20:22:10.280354 17618 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
I20260928 20:22:10.284186 17643 log.cc:824] T 2d7733443bac4a7cb397a65e91007560 P 95fe79ad6abc4659bb584e646111be16: Log is configured to *not* fsync() on all Append() calls
I20260928 20:22:10.285712 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 17618
I20260928 20:22:10.286001 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.3:42799
--local_ip_for_outbound_sockets=127.3.18.3
--tserver_master_addrs=127.3.18.62:43799
--webserver_port=33697
--webserver_interface=127.3.18.3
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab
--principal=kudu/127.3.18.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:45035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 20:22:10.286928 17643 tablet_bootstrap.cc:492] T 2d7733443bac4a7cb397a65e91007560 P 95fe79ad6abc4659bb584e646111be16: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 20:22:10.287331 17643 tablet_bootstrap.cc:492] T 2d7733443bac4a7cb397a65e91007560 P 95fe79ad6abc4659bb584e646111be16: Bootstrap complete.
Sep 28 20:22:10 dist-test-slave-b83v krb5kdc[16843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626930, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.2@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:22:10.289127 17643 ts_tablet_manager.cc:1397] T 2d7733443bac4a7cb397a65e91007560 P 95fe79ad6abc4659bb584e646111be16: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.006s
I20260928 20:22:10.290400 17643 raft_consensus.cc:348] T 2d7733443bac4a7cb397a65e91007560 P 95fe79ad6abc4659bb584e646111be16 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95fe79ad6abc4659bb584e646111be16" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 39927 } } peers { permanent_uuid: "3774a634d815476a80a6b57ac76edfd3" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34351 } } peers { permanent_uuid: "f8cdfd447c5d4a09bdcc7cce530c7927" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 42799 } }
I20260928 20:22:10.290668 17643 raft_consensus.cc:729] T 2d7733443bac4a7cb397a65e91007560 P 95fe79ad6abc4659bb584e646111be16 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95fe79ad6abc4659bb584e646111be16, State: Initialized, Role: FOLLOWER
I20260928 20:22:10.290843 17643 consensus_queue.cc:260] T 2d7733443bac4a7cb397a65e91007560 P 95fe79ad6abc4659bb584e646111be16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95fe79ad6abc4659bb584e646111be16" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 39927 } } peers { permanent_uuid: "3774a634d815476a80a6b57ac76edfd3" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34351 } } peers { permanent_uuid: "f8cdfd447c5d4a09bdcc7cce530c7927" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 42799 } }
I20260928 20:22:10.291200 17643 ts_tablet_manager.cc:1428] T 2d7733443bac4a7cb397a65e91007560 P 95fe79ad6abc4659bb584e646111be16: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260928 20:22:10.292943 17753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:10.281101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:43799 (local address 127.3.18.2:36463)
0928 20:22:10.281304 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:10.281365 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 20:22:10.282173 (+ 808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:22:10.282476 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:22:10.282481 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:22:10.282981 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:22:10.283323 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:10.283334 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:10.284206 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:10.284207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:10.285401 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:10.285404 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:10.286028 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:10.286030 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:10.286485 (+ 455us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:10.287199 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:22:10.287210 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:22:10.288962 (+ 1752us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:22:10.291105 (+ 2143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:22:10.291111 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:22:10.291113 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:22:10.291421 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:22:10.291739 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:22:10.291740 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:22:10.291740 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:22:10.291954 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:22:10.292301 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:22:10.292307 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:22:10.292543 (+ 236us) client_negotiation.cc:769] Sending connection context
0928 20:22:10.292653 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 20:22:10.292788 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":116,"threads_started":1}
I20260928 20:22:10.292956 17615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:10.281169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:36463 (local address 127.3.18.62:43799)
0928 20:22:10.281261 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:10.281263 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:10.281479 (+ 216us) server_negotiation.cc:407] Connection header received
0928 20:22:10.282310 (+ 831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:10.282311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:10.282338 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:10.282394 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:22:10.283453 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:10.284095 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:10.285508 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:10.285905 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:10.285996 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:10.289081 (+ 3085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:22:10.289082 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:22:10.289082 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:22:10.289090 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:22:10.290991 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:10.291525 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:10.291527 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:10.291528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:10.291607 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:10.292046 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:10.292048 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:10.292048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:10.292219 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:22:10.292304 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:10.292695 (+ 391us) server_negotiation.cc:299] Negotiation successful
0928 20:22:10.292807 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 20:22:10.293423 17751 heartbeater.cc:347] Connected to a master server at 127.3.18.62:43799
I20260928 20:22:10.293550 17751 heartbeater.cc:464] Registering TS with master...
I20260928 20:22:10.293720 17751 heartbeater.cc:511] Master 127.3.18.62:43799 requested a full tablet report, sending...
I20260928 20:22:10.294287 17424 ts_manager.cc:194] Registered new tserver with Master: 95fe79ad6abc4659bb584e646111be16 (127.3.18.2:39927)
I20260928 20:22:10.294898 17424 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.18.2@KRBTEST.COM'} at 127.3.18.2:36463
I20260928 20:22:10.295841 17751 heartbeater.cc:503] Master 127.3.18.62:43799 was elected leader, sending a full tablet report...
Sep 28 20:22:10 dist-test-slave-b83v krb5kdc[16843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626929, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.62@KRBTEST.COM for kudu/127.3.18.2@KRBTEST.COM
I20260928 20:22:10.303107 17757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:10.294755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48912 (local address 127.3.18.2:39927)
0928 20:22:10.294919 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:10.294925 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:10.294936 (+ 11us) server_negotiation.cc:407] Connection header received
0928 20:22:10.294974 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:10.294976 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:10.294995 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:10.295125 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:22:10.295592 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:10.296583 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:10.297273 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:10.297574 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:10.297661 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:10.300001 (+ 2340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:22:10.300008 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:22:10.300009 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:22:10.300015 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:22:10.301633 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:10.302118 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:10.302121 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:10.302122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:10.302181 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:10.302491 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:10.302492 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:10.302492 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:10.302657 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:22:10.302727 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:10.302922 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 20:22:10.303001 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":82,"threads_started":1}
I20260928 20:22:10.303108 17619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:10.294584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.2:39927 (local address 127.0.0.1:48912)
0928 20:22:10.294637 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:10.294647 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 20:22:10.294681 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:22:10.295127 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:22:10.295128 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:22:10.295460 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:22:10.295494 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:10.295496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:10.296664 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:10.296665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:10.297174 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:10.297176 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:10.297661 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:10.297664 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:10.297724 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:10.298289 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:22:10.298293 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:22:10.299859 (+ 1566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:22:10.301718 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:22:10.301728 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:22:10.301730 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:22:10.302002 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:22:10.302324 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:22:10.302325 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:22:10.302326 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:22:10.302411 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:22:10.302737 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:22:10.302747 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:22:10.302806 (+ 59us) client_negotiation.cc:769] Sending connection context
0928 20:22:10.302885 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 20:22:10.303006 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 20:22:10.303573 17685 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d7733443bac4a7cb397a65e91007560 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:22:09 UTC) from {username='kudu', principal='kudu/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:48912
I20260928 20:22:10.304208 17758 tablet_replica.cc:330] stopping tablet replica
I20260928 20:22:10.304342 17758 raft_consensus.cc:2229] T 2d7733443bac4a7cb397a65e91007560 P 95fe79ad6abc4659bb584e646111be16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:22:10.304436 17758 raft_consensus.cc:2258] T 2d7733443bac4a7cb397a65e91007560 P 95fe79ad6abc4659bb584e646111be16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:22:10.304742 17758 ts_tablet_manager.cc:1905] T 2d7733443bac4a7cb397a65e91007560 P 95fe79ad6abc4659bb584e646111be16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:22:10.306687 17758 ts_tablet_manager.cc:1918] T 2d7733443bac4a7cb397a65e91007560 P 95fe79ad6abc4659bb584e646111be16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:22:10.306823 17758 log.cc:1196] T 2d7733443bac4a7cb397a65e91007560 P 95fe79ad6abc4659bb584e646111be16: Deleting WAL directory at /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-1/wal/wals/2d7733443bac4a7cb397a65e91007560
I20260928 20:22:10.307065 17758 ts_tablet_manager.cc:1939] T 2d7733443bac4a7cb397a65e91007560 P 95fe79ad6abc4659bb584e646111be16: Deleting consensus metadata
I20260928 20:22:10.307739 17408 catalog_manager.cc:4841] TS 95fe79ad6abc4659bb584e646111be16 (127.3.18.2:39927): tablet 2d7733443bac4a7cb397a65e91007560 (table test-table [id=81284af5b52c45fda173e949de1b2f83]) successfully deleted
W20260928 20:22:10.377350 17755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:22:10.377581 17755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:22:10.377606 17755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:22:10.378782 17755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:22:10.378830 17755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:22:10.378856 17755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.3
I20260928 20:22:10.380390 17755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:45035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab
--principal=kudu/127.3.18.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.3:42799
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.18.3
--webserver_port=33697
--tserver_master_addrs=127.3.18.62:43799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.3
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:22:10.380719 17755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:22:10.380968 17755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:22:10.386317 17763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:22:10.386379 17764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:22:10.386341 17766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:22:10.386672 17755 server_base.cc:1034] running on GCE node
I20260928 20:22:10.386907 17755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:22:10.387183 17755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:22:10.388355 17755 hybrid_clock.cc:648] HybridClock initialized: now 1790626930388356 us; error 51 us; skew 500 ppm
Sep 28 20:22:10 dist-test-slave-b83v krb5kdc[16843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626930, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:22:10.391002 17755 init.cc:377] Logged in from keytab as kudu/127.3.18.3@KRBTEST.COM (short username kudu)
I20260928 20:22:10.391404 17755 webserver.cc:460] Webserver started at http://127.3.18.3:33697/ using document root <none> and password file <none>
I20260928 20:22:10.391654 17755 fs_manager.cc:359] Metadata directory not provided
I20260928 20:22:10.391714 17755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:22:10.393204 17755 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:22:10.393899 17773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:22:10.394120 17755 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:22:10.394212 17755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-2/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-2/wal
uuid: "f8cdfd447c5d4a09bdcc7cce530c7927"
format_stamp: "Formatted at 2026-09-28 20:22:09 on dist-test-slave-b83v"
server_key: "00013873962e4a4a6f8c35699921c4c9"
server_key_iv: "631a85baaef20a20468698ebd11638a6"
server_key_version: "encryptionkey@0"
I20260928 20:22:10.394563 17755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:22:10.404474 17755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:22:10.406972 17755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:22:10.407150 17755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:22:10.407419 17755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:22:10.407814 17755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:22:10.407863 17755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:22:10.407887 17755 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:22:10.407900 17755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:22:10.413770 17755 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.3:42799
I20260928 20:22:10.413810 17886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.3:42799 every 8 connection(s)
I20260928 20:22:10.414166 17755 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
Sep 28 20:22:10 dist-test-slave-b83v krb5kdc[16843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626930, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.3@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:22:10.423030 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 17755
I20260928 20:22:10.423862 17615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:10.414921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:41775 (local address 127.3.18.62:43799)
0928 20:22:10.415013 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:10.415015 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:10.415200 (+ 185us) server_negotiation.cc:407] Connection header received
0928 20:22:10.415729 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:10.415730 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:10.415740 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:10.415810 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:22:10.416768 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:10.417275 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:10.418281 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:10.418461 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:10.418517 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:10.420714 (+ 2197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:22:10.420716 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:22:10.420716 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:22:10.420722 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:22:10.422028 (+ 1306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:10.422443 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:10.422444 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:10.422445 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:10.422502 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:10.422795 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:10.422796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:10.422797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:10.422998 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:22:10.423137 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:10.423628 (+ 491us) server_negotiation.cc:299] Negotiation successful
0928 20:22:10.423731 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 20:22:10.423964 17889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:10.414882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:43799 (local address 127.3.18.3:41775)
0928 20:22:10.415077 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:10.415114 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 20:22:10.415639 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:22:10.415893 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:22:10.415896 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:22:10.416476 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:22:10.416676 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:10.416682 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:10.417356 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:10.417357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:10.418198 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:10.418200 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:10.418572 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:10.418573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:10.418679 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:10.419156 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:22:10.419164 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:22:10.420624 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:22:10.422131 (+ 1507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:22:10.422138 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:22:10.422139 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:22:10.422357 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:22:10.422607 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:22:10.422608 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:22:10.422608 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:22:10.422707 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:22:10.423086 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:22:10.423119 (+ 33us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:22:10.423312 (+ 193us) client_negotiation.cc:769] Sending connection context
0928 20:22:10.423432 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 20:22:10.423657 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":104,"threads_started":1}
I20260928 20:22:10.424476 17887 heartbeater.cc:347] Connected to a master server at 127.3.18.62:43799
I20260928 20:22:10.424671 17887 heartbeater.cc:464] Registering TS with master...
I20260928 20:22:10.424875 17887 heartbeater.cc:511] Master 127.3.18.62:43799 requested a full tablet report, sending...
I20260928 20:22:10.425316 17424 ts_manager.cc:194] Registered new tserver with Master: f8cdfd447c5d4a09bdcc7cce530c7927 (127.3.18.3:42799)
I20260928 20:22:10.425806 17424 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.18.3@KRBTEST.COM'} at 127.3.18.3:41775
I20260928 20:22:10.430646 17891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:10.423449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53816 (local address 127.3.18.62:43799)
0928 20:22:10.423641 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:10.423643 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:10.423677 (+ 34us) server_negotiation.cc:407] Connection header received
0928 20:22:10.423810 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:10.423811 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:10.423824 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:10.423892 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:22:10.424598 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:10.425337 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:10.426330 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:10.426541 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:10.426614 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:10.428034 (+ 1420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:22:10.428036 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:22:10.428036 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:22:10.428042 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:22:10.429385 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:10.429819 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:10.429821 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:10.429822 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:10.429885 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:10.430091 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:10.430092 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:10.430092 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:10.430218 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:22:10.430418 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:10.430454 (+ 36us) server_negotiation.cc:299] Negotiation successful
0928 20:22:10.430535 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":82,"threads_started":1}
I20260928 20:22:10.431429 3144 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 20:22:10.434360 17891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:10.431927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53820 (local address 127.3.18.62:43799)
0928 20:22:10.432001 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:10.432006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:10.432103 (+ 97us) server_negotiation.cc:407] Connection header received
0928 20:22:10.432159 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:10.432160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:10.432197 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:10.432274 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:22:10.432411 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:10.432911 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:10.433711 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:10.433910 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:10.433971 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:10.434095 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:22:10.434196 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:22:10.434256 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:10.434287 (+ 31us) server_negotiation.cc:299] Negotiation successful
0928 20:22:10.434307 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 20:22:10.434680 17424 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:53820:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 20:22:10.436458 17424 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 20:22:10.439152 17685 tablet_service.cc:1467] Processing CreateTablet for tablet 246f5099c79d44cc9bbd11e6dbdcb9f4 (DEFAULT_TABLE table=test-table [id=0a831278592c45bbbda24fe51f931fa3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:22:10.439147 17546 tablet_service.cc:1467] Processing CreateTablet for tablet 246f5099c79d44cc9bbd11e6dbdcb9f4 (DEFAULT_TABLE table=test-table [id=0a831278592c45bbbda24fe51f931fa3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:22:10.439355 17685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 246f5099c79d44cc9bbd11e6dbdcb9f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:22:10.439359 17546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 246f5099c79d44cc9bbd11e6dbdcb9f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:22:10.440737 17504 tablet_bootstrap.cc:492] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 3774a634d815476a80a6b57ac76edfd3: Bootstrap starting.
I20260928 20:22:10.440932 17643 tablet_bootstrap.cc:492] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 95fe79ad6abc4659bb584e646111be16: Bootstrap starting.
I20260928 20:22:10.441097 17504 tablet_bootstrap.cc:654] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 3774a634d815476a80a6b57ac76edfd3: Neither blocks nor log segments found. Creating new log.
I20260928 20:22:10.441254 17643 tablet_bootstrap.cc:654] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 95fe79ad6abc4659bb584e646111be16: Neither blocks nor log segments found. Creating new log.
I20260928 20:22:10.441661 17504 tablet_bootstrap.cc:492] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 3774a634d815476a80a6b57ac76edfd3: No bootstrap required, opened a new log
I20260928 20:22:10.441702 17504 ts_tablet_manager.cc:1397] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 3774a634d815476a80a6b57ac76edfd3: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 20:22:10.441797 17504 raft_consensus.cc:348] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 3774a634d815476a80a6b57ac76edfd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8cdfd447c5d4a09bdcc7cce530c7927" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 42799 } } peers { permanent_uuid: "3774a634d815476a80a6b57ac76edfd3" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34351 } } peers { permanent_uuid: "95fe79ad6abc4659bb584e646111be16" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 39927 } }
I20260928 20:22:10.441855 17504 raft_consensus.cc:374] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 3774a634d815476a80a6b57ac76edfd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:22:10.441901 17504 raft_consensus.cc:729] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 3774a634d815476a80a6b57ac76edfd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3774a634d815476a80a6b57ac76edfd3, State: Initialized, Role: FOLLOWER
I20260928 20:22:10.441952 17504 consensus_queue.cc:260] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 3774a634d815476a80a6b57ac76edfd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8cdfd447c5d4a09bdcc7cce530c7927" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 42799 } } peers { permanent_uuid: "3774a634d815476a80a6b57ac76edfd3" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34351 } } peers { permanent_uuid: "95fe79ad6abc4659bb584e646111be16" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 39927 } }
I20260928 20:22:10.442032 17504 ts_tablet_manager.cc:1428] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 3774a634d815476a80a6b57ac76edfd3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 20:22:10.442221 17643 tablet_bootstrap.cc:492] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 95fe79ad6abc4659bb584e646111be16: No bootstrap required, opened a new log
I20260928 20:22:10.442263 17643 ts_tablet_manager.cc:1397] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 95fe79ad6abc4659bb584e646111be16: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260928 20:22:10.442649 17643 raft_consensus.cc:348] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 95fe79ad6abc4659bb584e646111be16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8cdfd447c5d4a09bdcc7cce530c7927" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 42799 } } peers { permanent_uuid: "3774a634d815476a80a6b57ac76edfd3" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34351 } } peers { permanent_uuid: "95fe79ad6abc4659bb584e646111be16" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 39927 } }
I20260928 20:22:10.442731 17643 raft_consensus.cc:374] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 95fe79ad6abc4659bb584e646111be16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:22:10.442777 17643 raft_consensus.cc:729] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 95fe79ad6abc4659bb584e646111be16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95fe79ad6abc4659bb584e646111be16, State: Initialized, Role: FOLLOWER
I20260928 20:22:10.442822 17643 consensus_queue.cc:260] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 95fe79ad6abc4659bb584e646111be16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8cdfd447c5d4a09bdcc7cce530c7927" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 42799 } } peers { permanent_uuid: "3774a634d815476a80a6b57ac76edfd3" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34351 } } peers { permanent_uuid: "95fe79ad6abc4659bb584e646111be16" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 39927 } }
I20260928 20:22:10.442909 17643 ts_tablet_manager.cc:1428] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 95fe79ad6abc4659bb584e646111be16: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 20:22:10.443512 17619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:10.438832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.3:42799 (local address 127.0.0.1:37602)
0928 20:22:10.438925 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:10.438935 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 20:22:10.438971 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:22:10.439537 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:22:10.439538 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:22:10.439540 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:22:10.439596 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:10.439597 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:10.441100 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:10.441101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:10.442195 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:10.442198 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:10.443262 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:10.443263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:10.443326 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:10.443329 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 20:22:10.443389 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 20:22:10.443434 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260928 20:22:10.443523 17893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:10.439065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37602 (local address 127.3.18.3:42799)
0928 20:22:10.439304 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:10.439309 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:10.439321 (+ 12us) server_negotiation.cc:407] Connection header received
0928 20:22:10.439360 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:10.439363 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:10.439380 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:10.439516 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:22:10.439707 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:10.440908 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:10.442372 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:10.443172 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:10.443264 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:10.443330 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:10.443404 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 20:22:10.443443 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":114,"threads_started":1}
I20260928 20:22:10.444329 17821 tablet_service.cc:1467] Processing CreateTablet for tablet 246f5099c79d44cc9bbd11e6dbdcb9f4 (DEFAULT_TABLE table=test-table [id=0a831278592c45bbbda24fe51f931fa3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:22:10.444676 17821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 246f5099c79d44cc9bbd11e6dbdcb9f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:22:10.447732 17896 tablet_bootstrap.cc:492] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P f8cdfd447c5d4a09bdcc7cce530c7927: Bootstrap starting.
I20260928 20:22:10.448586 17896 tablet_bootstrap.cc:654] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P f8cdfd447c5d4a09bdcc7cce530c7927: Neither blocks nor log segments found. Creating new log.
I20260928 20:22:10.449085 17896 log.cc:824] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P f8cdfd447c5d4a09bdcc7cce530c7927: Log is configured to *not* fsync() on all Append() calls
I20260928 20:22:10.449888 17896 tablet_bootstrap.cc:492] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P f8cdfd447c5d4a09bdcc7cce530c7927: No bootstrap required, opened a new log
I20260928 20:22:10.449987 17896 ts_tablet_manager.cc:1397] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P f8cdfd447c5d4a09bdcc7cce530c7927: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 20:22:10.452081 17896 raft_consensus.cc:348] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P f8cdfd447c5d4a09bdcc7cce530c7927 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8cdfd447c5d4a09bdcc7cce530c7927" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 42799 } } peers { permanent_uuid: "3774a634d815476a80a6b57ac76edfd3" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34351 } } peers { permanent_uuid: "95fe79ad6abc4659bb584e646111be16" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 39927 } }
I20260928 20:22:10.452279 17896 raft_consensus.cc:374] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P f8cdfd447c5d4a09bdcc7cce530c7927 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:22:10.452327 17896 raft_consensus.cc:729] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P f8cdfd447c5d4a09bdcc7cce530c7927 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8cdfd447c5d4a09bdcc7cce530c7927, State: Initialized, Role: FOLLOWER
I20260928 20:22:10.452491 17896 consensus_queue.cc:260] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P f8cdfd447c5d4a09bdcc7cce530c7927 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8cdfd447c5d4a09bdcc7cce530c7927" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 42799 } } peers { permanent_uuid: "3774a634d815476a80a6b57ac76edfd3" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34351 } } peers { permanent_uuid: "95fe79ad6abc4659bb584e646111be16" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 39927 } }
I20260928 20:22:10.452800 17896 ts_tablet_manager.cc:1428] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P f8cdfd447c5d4a09bdcc7cce530c7927: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 20:22:10.452857 17887 heartbeater.cc:503] Master 127.3.18.62:43799 was elected leader, sending a full tablet report...
W20260928 20:22:10.531021 17752 tablet.cc:2369] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 95fe79ad6abc4659bb584e646111be16: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 20:22:10.629694 17613 tablet.cc:2369] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 3774a634d815476a80a6b57ac76edfd3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 20:22:10.664778 17888 tablet.cc:2369] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P f8cdfd447c5d4a09bdcc7cce530c7927: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 20:22:10.757845 17756 raft_consensus.cc:482] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 95fe79ad6abc4659bb584e646111be16 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:22:10.758021 17756 raft_consensus.cc:504] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 95fe79ad6abc4659bb584e646111be16 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8cdfd447c5d4a09bdcc7cce530c7927" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 42799 } } peers { permanent_uuid: "3774a634d815476a80a6b57ac76edfd3" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34351 } } peers { permanent_uuid: "95fe79ad6abc4659bb584e646111be16" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 39927 } }
I20260928 20:22:10.758457 17756 leader_election.cc:288] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 95fe79ad6abc4659bb584e646111be16 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8cdfd447c5d4a09bdcc7cce530c7927 (127.3.18.3:42799), 3774a634d815476a80a6b57ac76edfd3 (127.3.18.1:34351)
I20260928 20:22:10.762207 17893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:10.758908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:59543 (local address 127.3.18.3:42799)
0928 20:22:10.759012 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:10.759015 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:10.759027 (+ 12us) server_negotiation.cc:407] Connection header received
0928 20:22:10.759062 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:10.759063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:10.759077 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:10.759140 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:22:10.759376 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:10.760061 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:10.761283 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:10.761804 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:10.761859 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:10.761867 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:10.762113 (+ 246us) server_negotiation.cc:299] Negotiation successful
0928 20:22:10.762150 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 20:22:10.762238 17753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:10.758738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.3:42799 (local address 127.3.18.2:59543)
0928 20:22:10.758800 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:10.758813 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 20:22:10.758877 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:22:10.759158 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:22:10.759159 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:22:10.759162 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:22:10.759249 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:10.759251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:10.760164 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:10.760165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:10.761175 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:10.761177 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:10.761877 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:10.761878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:10.761966 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:10.761999 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 20:22:10.762100 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 20:22:10.762148 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 20:22:10.762722 17900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:10.759032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:46103 (local address 127.3.18.1:34351)
0928 20:22:10.759214 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:10.759216 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:10.759225 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:22:10.759251 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:10.759252 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:10.759265 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:10.759333 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:22:10.759724 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:10.760497 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:10.761678 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:10.762224 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:10.762329 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:10.762344 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:10.762634 (+ 290us) server_negotiation.cc:299] Negotiation successful
0928 20:22:10.762671 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":75,"threads_started":1}
I20260928 20:22:10.762766 17899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:10.758867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.1:34351 (local address 127.3.18.2:46103)
0928 20:22:10.759006 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:10.759047 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 20:22:10.759117 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:22:10.759374 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:22:10.759375 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:22:10.759378 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:22:10.759463 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:10.759464 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:10.760599 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:10.760600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:10.761585 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:10.761588 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:10.762430 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:10.762431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:10.762539 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:10.762545 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 20:22:10.762630 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 20:22:10.762673 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":61,"threads_started":1}
I20260928 20:22:10.762725 17841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "246f5099c79d44cc9bbd11e6dbdcb9f4" candidate_uuid: "95fe79ad6abc4659bb584e646111be16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8cdfd447c5d4a09bdcc7cce530c7927" is_pre_election: true
I20260928 20:22:10.762907 17841 raft_consensus.cc:2454] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P f8cdfd447c5d4a09bdcc7cce530c7927 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95fe79ad6abc4659bb584e646111be16 in term 0.
I20260928 20:22:10.763159 17639 leader_election.cc:302] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 95fe79ad6abc4659bb584e646111be16 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95fe79ad6abc4659bb584e646111be16, f8cdfd447c5d4a09bdcc7cce530c7927; no voters:
I20260928 20:22:10.763136 17566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "246f5099c79d44cc9bbd11e6dbdcb9f4" candidate_uuid: "95fe79ad6abc4659bb584e646111be16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3774a634d815476a80a6b57ac76edfd3" is_pre_election: true
I20260928 20:22:10.763310 17566 raft_consensus.cc:2454] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 3774a634d815476a80a6b57ac76edfd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95fe79ad6abc4659bb584e646111be16 in term 0.
I20260928 20:22:10.763314 17756 raft_consensus.cc:2781] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 95fe79ad6abc4659bb584e646111be16 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 20:22:10.763376 17756 raft_consensus.cc:482] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 95fe79ad6abc4659bb584e646111be16 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 20:22:10.763404 17756 raft_consensus.cc:3037] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 95fe79ad6abc4659bb584e646111be16 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:22:10.764191 17756 raft_consensus.cc:504] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 95fe79ad6abc4659bb584e646111be16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8cdfd447c5d4a09bdcc7cce530c7927" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 42799 } } peers { permanent_uuid: "3774a634d815476a80a6b57ac76edfd3" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34351 } } peers { permanent_uuid: "95fe79ad6abc4659bb584e646111be16" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 39927 } }
I20260928 20:22:10.764365 17756 leader_election.cc:288] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 95fe79ad6abc4659bb584e646111be16 [CANDIDATE]: Term 1 election: Requested vote from peers f8cdfd447c5d4a09bdcc7cce530c7927 (127.3.18.3:42799), 3774a634d815476a80a6b57ac76edfd3 (127.3.18.1:34351)
I20260928 20:22:10.764555 17841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "246f5099c79d44cc9bbd11e6dbdcb9f4" candidate_uuid: "95fe79ad6abc4659bb584e646111be16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8cdfd447c5d4a09bdcc7cce530c7927"
I20260928 20:22:10.764570 17566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "246f5099c79d44cc9bbd11e6dbdcb9f4" candidate_uuid: "95fe79ad6abc4659bb584e646111be16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3774a634d815476a80a6b57ac76edfd3"
I20260928 20:22:10.764665 17566 raft_consensus.cc:3037] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 3774a634d815476a80a6b57ac76edfd3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:22:10.764665 17841 raft_consensus.cc:3037] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P f8cdfd447c5d4a09bdcc7cce530c7927 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:22:10.765573 17566 raft_consensus.cc:2454] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 3774a634d815476a80a6b57ac76edfd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95fe79ad6abc4659bb584e646111be16 in term 1.
I20260928 20:22:10.765690 17841 raft_consensus.cc:2454] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P f8cdfd447c5d4a09bdcc7cce530c7927 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95fe79ad6abc4659bb584e646111be16 in term 1.
I20260928 20:22:10.765837 17639 leader_election.cc:302] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 95fe79ad6abc4659bb584e646111be16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95fe79ad6abc4659bb584e646111be16, f8cdfd447c5d4a09bdcc7cce530c7927; no voters:
I20260928 20:22:10.766007 17756 raft_consensus.cc:2781] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 95fe79ad6abc4659bb584e646111be16 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:22:10.766219 17756 raft_consensus.cc:686] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 95fe79ad6abc4659bb584e646111be16 [term 1 LEADER]: Becoming Leader. State: Replica: 95fe79ad6abc4659bb584e646111be16, State: Running, Role: LEADER
I20260928 20:22:10.766283 17756 consensus_queue.cc:237] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 95fe79ad6abc4659bb584e646111be16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8cdfd447c5d4a09bdcc7cce530c7927" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 42799 } } peers { permanent_uuid: "3774a634d815476a80a6b57ac76edfd3" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34351 } } peers { permanent_uuid: "95fe79ad6abc4659bb584e646111be16" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 39927 } }
I20260928 20:22:10.767192 17424 catalog_manager.cc:5495] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 95fe79ad6abc4659bb584e646111be16 reported cstate change: term changed from 0 to 1, leader changed from <none> to 95fe79ad6abc4659bb584e646111be16 (127.3.18.2). New cstate: current_term: 1 leader_uuid: "95fe79ad6abc4659bb584e646111be16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8cdfd447c5d4a09bdcc7cce530c7927" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 42799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3774a634d815476a80a6b57ac76edfd3" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95fe79ad6abc4659bb584e646111be16" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 39927 } health_report { overall_health: HEALTHY } } }
I20260928 20:22:10.803283 17757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:10.800797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48920 (local address 127.3.18.2:39927)
0928 20:22:10.800853 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:10.800855 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:10.800881 (+ 26us) server_negotiation.cc:407] Connection header received
0928 20:22:10.800927 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:10.800928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:10.800951 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:10.801028 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:22:10.801151 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:10.801660 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:10.802447 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:10.802671 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:10.802752 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:10.802912 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:22:10.803025 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:22:10.803117 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:10.803157 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 20:22:10.803179 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 20:22:10.805105 17566 raft_consensus.cc:1261] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 3774a634d815476a80a6b57ac76edfd3 [term 1 FOLLOWER]: Refusing update from remote peer 95fe79ad6abc4659bb584e646111be16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:22:10.805123 17841 raft_consensus.cc:1261] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P f8cdfd447c5d4a09bdcc7cce530c7927 [term 1 FOLLOWER]: Refusing update from remote peer 95fe79ad6abc4659bb584e646111be16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:22:10.805466 17901 consensus_queue.cc:1035] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 95fe79ad6abc4659bb584e646111be16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8cdfd447c5d4a09bdcc7cce530c7927" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 42799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:22:10.805636 17756 consensus_queue.cc:1035] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 95fe79ad6abc4659bb584e646111be16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3774a634d815476a80a6b57ac76edfd3" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:22:10.807708 17904 mvcc.cc:205] Tried to move back new op lower bound from 7334407908574478336 to 7334407908419657728. Current Snapshot: MvccSnapshot[applied={T|T < 7334407908574478336}]
I20260928 20:22:10.808560 17905 mvcc.cc:205] Tried to move back new op lower bound from 7334407908574478336 to 7334407908419657728. Current Snapshot: MvccSnapshot[applied={T|T < 7334407908574478336}]
I20260928 20:22:10.810128 17423 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:53820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:22:10.810232 17423 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:53820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:22:10.811020 17423 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bd585a8d7a1245e89eed0a0ea71abc14: Sending DeleteTablet for 3 replicas of tablet 246f5099c79d44cc9bbd11e6dbdcb9f4
I20260928 20:22:10.811412 17546 tablet_service.cc:1514] Processing DeleteTablet for tablet 246f5099c79d44cc9bbd11e6dbdcb9f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:22:10 UTC) from {username='kudu', principal='kudu/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:56914
I20260928 20:22:10.811448 17821 tablet_service.cc:1514] Processing DeleteTablet for tablet 246f5099c79d44cc9bbd11e6dbdcb9f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:22:10 UTC) from {username='kudu', principal='kudu/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:37602
I20260928 20:22:10.811688 17915 tablet_replica.cc:330] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P f8cdfd447c5d4a09bdcc7cce530c7927: stopping tablet replica
I20260928 20:22:10.811700 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 17462
I20260928 20:22:10.812669 17685 tablet_service.cc:1514] Processing DeleteTablet for tablet 246f5099c79d44cc9bbd11e6dbdcb9f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:22:10 UTC) from {username='kudu', principal='kudu/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:48912
I20260928 20:22:10.812881 17916 tablet_replica.cc:330] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 95fe79ad6abc4659bb584e646111be16: stopping tablet replica
I20260928 20:22:10.812971 17916 raft_consensus.cc:2229] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 95fe79ad6abc4659bb584e646111be16 [term 1 LEADER]: Raft consensus shutting down.
I20260928 20:22:10.813095 17916 raft_consensus.cc:2258] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 95fe79ad6abc4659bb584e646111be16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:22:10.813495 17916 ts_tablet_manager.cc:1905] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 95fe79ad6abc4659bb584e646111be16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:22:10.815212 17916 ts_tablet_manager.cc:1918] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 95fe79ad6abc4659bb584e646111be16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:22:10.815305 17916 log.cc:1196] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 95fe79ad6abc4659bb584e646111be16: Deleting WAL directory at /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-1/wal/wals/246f5099c79d44cc9bbd11e6dbdcb9f4
I20260928 20:22:10.815563 17916 ts_tablet_manager.cc:1939] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 95fe79ad6abc4659bb584e646111be16: Deleting consensus metadata
I20260928 20:22:10.815987 17408 catalog_manager.cc:4841] TS 95fe79ad6abc4659bb584e646111be16 (127.3.18.2:39927): tablet 246f5099c79d44cc9bbd11e6dbdcb9f4 (table test-table [id=0a831278592c45bbbda24fe51f931fa3]) successfully deleted
I20260928 20:22:10.816488 17915 raft_consensus.cc:2229] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P f8cdfd447c5d4a09bdcc7cce530c7927 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:22:10.816659 17915 raft_consensus.cc:2258] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P f8cdfd447c5d4a09bdcc7cce530c7927 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:22:10.817193 17915 ts_tablet_manager.cc:1905] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P f8cdfd447c5d4a09bdcc7cce530c7927: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 20:22:10.817546 17410 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.18.1:34351 (error 108)
W20260928 20:22:10.818691 17619 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:22:10.818040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.1:34351 (local address 127.0.0.1:56928)
0928 20:22:10.818089 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:10.818100 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:22:10.818140 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:22:10.818639 (+ 499us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.1:34351: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 20:22:10.818889 17915 ts_tablet_manager.cc:1918] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P f8cdfd447c5d4a09bdcc7cce530c7927: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:22:10.818975 17915 log.cc:1196] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P f8cdfd447c5d4a09bdcc7cce530c7927: Deleting WAL directory at /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-2/wal/wals/246f5099c79d44cc9bbd11e6dbdcb9f4
W20260928 20:22:10.818812 17410 catalog_manager.cc:4584] TS 3774a634d815476a80a6b57ac76edfd3 (127.3.18.1:34351): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 246f5099c79d44cc9bbd11e6dbdcb9f4: Network error: Client connection negotiation failed: client connection to 127.3.18.1:34351: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 20:22:10.819279 17915 ts_tablet_manager.cc:1939] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P f8cdfd447c5d4a09bdcc7cce530c7927: Deleting consensus metadata
I20260928 20:22:10.819665 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 17618
I20260928 20:22:10.819734 17410 catalog_manager.cc:4841] TS f8cdfd447c5d4a09bdcc7cce530c7927 (127.3.18.3:42799): tablet 246f5099c79d44cc9bbd11e6dbdcb9f4 (table test-table [id=0a831278592c45bbbda24fe51f931fa3]) successfully deleted
I20260928 20:22:10.826655 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 17755
I20260928 20:22:10.831472 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 17391
I20260928 20:22:10.837157 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:43799
--webserver_interface=127.3.18.62
--webserver_port=37001
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:45035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.18.62:43799
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:22:10.923628 17918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:22:10.923924 17918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:22:10.923965 17918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:22:10.925122 17918 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:22:10.925194 17918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:22:10.925218 17918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:22:10.925240 17918 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:22:10.925271 17918 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:22:10.926796 17918 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:45035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.18.62:43799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:43799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.18.62
--webserver_port=37001
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:22:10.927080 17918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:22:10.927331 17918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:22:10.933156 17924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:22:10.933199 17923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:22:10.933214 17918 server_base.cc:1034] running on GCE node
W20260928 20:22:10.933171 17926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:22:10.933614 17918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:22:10.933980 17918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:22:10.935202 17918 hybrid_clock.cc:648] HybridClock initialized: now 1790626930935186 us; error 49 us; skew 500 ppm
Sep 28 20:22:10 dist-test-slave-b83v krb5kdc[16843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626930, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:22:10.938073 17918 init.cc:377] Logged in from keytab as kudu/127.3.18.62@KRBTEST.COM (short username kudu)
I20260928 20:22:10.938673 17918 webserver.cc:460] Webserver started at http://127.3.18.62:37001/ using document root <none> and password file <none>
I20260928 20:22:10.938936 17918 fs_manager.cc:359] Metadata directory not provided
I20260928 20:22:10.939014 17918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:22:10.940583 17918 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260928 20:22:10.941426 17933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:22:10.941637 17918 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:22:10.941757 17918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/master-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/master-0/wal
uuid: "bd585a8d7a1245e89eed0a0ea71abc14"
format_stamp: "Formatted at 2026-09-28 20:22:09 on dist-test-slave-b83v"
server_key: "b67d379996b5570a2216fda6f0d9a5e6"
server_key_iv: "b81faaab8880c8a28e7c0a47e42710b3"
server_key_version: "encryptionkey@0"
I20260928 20:22:10.942222 17918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:22:10.967475 17918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:22:10.970909 17918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:22:10.971113 17918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:22:10.976233 17918 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.62:43799
I20260928 20:22:10.976302 17985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.62:43799 every 8 connection(s)
I20260928 20:22:10.976773 17918 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
I20260928 20:22:10.978714 17986 sys_catalog.cc:263] Verifying existing consensus state
I20260928 20:22:10.979353 17986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd585a8d7a1245e89eed0a0ea71abc14: Bootstrap starting.
I20260928 20:22:10.982874 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 17918
I20260928 20:22:10.983181 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.1:34351
--local_ip_for_outbound_sockets=127.3.18.1
--tserver_master_addrs=127.3.18.62:43799
--webserver_port=41243
--webserver_interface=127.3.18.1
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab
--principal=kudu/127.3.18.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:45035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 20:22:10.988638 17986 log.cc:824] T 00000000000000000000000000000000 P bd585a8d7a1245e89eed0a0ea71abc14: Log is configured to *not* fsync() on all Append() calls
I20260928 20:22:10.990952 17986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd585a8d7a1245e89eed0a0ea71abc14: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 20:22:10.991283 17986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd585a8d7a1245e89eed0a0ea71abc14: Bootstrap complete.
I20260928 20:22:10.993678 17986 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd585a8d7a1245e89eed0a0ea71abc14 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd585a8d7a1245e89eed0a0ea71abc14" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 43799 } }
I20260928 20:22:10.993944 17986 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd585a8d7a1245e89eed0a0ea71abc14 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd585a8d7a1245e89eed0a0ea71abc14, State: Initialized, Role: FOLLOWER
I20260928 20:22:10.994182 17986 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd585a8d7a1245e89eed0a0ea71abc14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd585a8d7a1245e89eed0a0ea71abc14" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 43799 } }
I20260928 20:22:10.994282 17986 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd585a8d7a1245e89eed0a0ea71abc14 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:22:10.994338 17986 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd585a8d7a1245e89eed0a0ea71abc14 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:22:10.994387 17986 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd585a8d7a1245e89eed0a0ea71abc14 [term 2 FOLLOWER]: Advancing to term 3
I20260928 20:22:10.995452 17986 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd585a8d7a1245e89eed0a0ea71abc14 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd585a8d7a1245e89eed0a0ea71abc14" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 43799 } }
I20260928 20:22:10.995577 17986 leader_election.cc:302] T 00000000000000000000000000000000 P bd585a8d7a1245e89eed0a0ea71abc14 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd585a8d7a1245e89eed0a0ea71abc14; no voters:
I20260928 20:22:10.995811 17986 leader_election.cc:288] T 00000000000000000000000000000000 P bd585a8d7a1245e89eed0a0ea71abc14 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 20:22:10.995872 17990 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd585a8d7a1245e89eed0a0ea71abc14 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 20:22:10.996042 17990 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd585a8d7a1245e89eed0a0ea71abc14 [term 3 LEADER]: Becoming Leader. State: Replica: bd585a8d7a1245e89eed0a0ea71abc14, State: Running, Role: LEADER
I20260928 20:22:10.996145 17986 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd585a8d7a1245e89eed0a0ea71abc14 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:22:10.996264 17990 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd585a8d7a1245e89eed0a0ea71abc14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd585a8d7a1245e89eed0a0ea71abc14" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 43799 } }
I20260928 20:22:10.996750 17990 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd585a8d7a1245e89eed0a0ea71abc14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd585a8d7a1245e89eed0a0ea71abc14. Latest consensus state: current_term: 3 leader_uuid: "bd585a8d7a1245e89eed0a0ea71abc14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd585a8d7a1245e89eed0a0ea71abc14" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 43799 } } }
I20260928 20:22:10.996774 17992 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd585a8d7a1245e89eed0a0ea71abc14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "bd585a8d7a1245e89eed0a0ea71abc14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd585a8d7a1245e89eed0a0ea71abc14" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 43799 } } }
I20260928 20:22:10.996901 17992 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd585a8d7a1245e89eed0a0ea71abc14 [sys.catalog]: This master's current role is: LEADER
I20260928 20:22:10.997093 17990 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd585a8d7a1245e89eed0a0ea71abc14 [sys.catalog]: This master's current role is: LEADER
I20260928 20:22:10.997152 17999 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:22:10.998126 17999 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:22:10.998246 17999 catalog_manager.cc:1259] Loaded cluster ID: b14de213dd4c4000b5f505e94ede52e1
I20260928 20:22:10.998328 17999 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 20:22:10.998765 17999 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f1403a599ad google::LogMessage::Fail() at ??:0
@ 0x7f1403a5c88a google::LogMessage::SendToLog() at ??:0
@ 0x7f1403a594de google::LogMessage::Flush() at ??:0
@ 0x7f1403a5aa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f140b97e0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f140b9878b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f14059270d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f140591c954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f14067806db start_thread at ??:0
@ 0x7f140255761f clone at ??:0
W20260928 20:22:11.084918 17988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:22:11.085186 17988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:22:11.085230 17988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:22:11.086481 17988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:22:11.086560 17988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:22:11.086609 17988 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.1
I20260928 20:22:11.088341 17988 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:45035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab
--principal=kudu/127.3.18.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.1:34351
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.18.1
--webserver_port=41243
--tserver_master_addrs=127.3.18.62:43799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.1
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:22:11.088644 17988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:22:11.088891 17988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:22:11.094754 18025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:22:11.094805 18022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:22:11.094754 18023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:22:11.095062 17988 server_base.cc:1034] running on GCE node
I20260928 20:22:11.095461 17988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:22:11.095719 17988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:22:11.096928 17988 hybrid_clock.cc:648] HybridClock initialized: now 1790626931096917 us; error 41 us; skew 500 ppm
Sep 28 20:22:11 dist-test-slave-b83v krb5kdc[16843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626931, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:22:11.099606 17988 init.cc:377] Logged in from keytab as kudu/127.3.18.1@KRBTEST.COM (short username kudu)
I20260928 20:22:11.100253 17988 webserver.cc:460] Webserver started at http://127.3.18.1:41243/ using document root <none> and password file <none>
I20260928 20:22:11.100538 17988 fs_manager.cc:359] Metadata directory not provided
I20260928 20:22:11.100631 17988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:22:11.102414 17988 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 20:22:11.103221 18032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:22:11.103426 17988 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:22:11.103536 17988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-0/wal
uuid: "3774a634d815476a80a6b57ac76edfd3"
format_stamp: "Formatted at 2026-09-28 20:22:09 on dist-test-slave-b83v"
server_key: "12103322db7fb85c6d347caf8fc9547b"
server_key_iv: "4ed241b44599fc4a7a075a13efb90dde"
server_key_version: "encryptionkey@0"
I20260928 20:22:11.103960 17988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:22:11.130165 17988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:22:11.132890 17988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:22:11.133061 17988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:22:11.133327 17988 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:22:11.133816 18039 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 20:22:11.135572 17988 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 20:22:11.135689 17988 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 20:22:11.135792 17988 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 20:22:11.136639 17988 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 20:22:11.136708 17988 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 20:22:11.136806 18039 tablet_bootstrap.cc:492] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 3774a634d815476a80a6b57ac76edfd3: Bootstrap starting.
I20260928 20:22:11.145063 17988 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.1:34351
I20260928 20:22:11.145135 18146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.1:34351 every 8 connection(s)
I20260928 20:22:11.145644 17988 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
I20260928 20:22:11.146900 18039 log.cc:824] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 3774a634d815476a80a6b57ac76edfd3: Log is configured to *not* fsync() on all Append() calls
I20260928 20:22:11.148826 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 17988
I20260928 20:22:11.149183 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.2:39927
--local_ip_for_outbound_sockets=127.3.18.2
--tserver_master_addrs=127.3.18.62:43799
--webserver_port=33175
--webserver_interface=127.3.18.2
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab
--principal=kudu/127.3.18.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:45035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 20:22:11.149734 18039 tablet_bootstrap.cc:492] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 3774a634d815476a80a6b57ac76edfd3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 20:22:11.150028 18039 tablet_bootstrap.cc:492] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 3774a634d815476a80a6b57ac76edfd3: Bootstrap complete.
I20260928 20:22:11.150995 18039 ts_tablet_manager.cc:1397] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 3774a634d815476a80a6b57ac76edfd3: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.006s
I20260928 20:22:11.152252 18039 raft_consensus.cc:348] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 3774a634d815476a80a6b57ac76edfd3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8cdfd447c5d4a09bdcc7cce530c7927" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 42799 } } peers { permanent_uuid: "3774a634d815476a80a6b57ac76edfd3" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34351 } } peers { permanent_uuid: "95fe79ad6abc4659bb584e646111be16" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 39927 } }
I20260928 20:22:11.152534 18039 raft_consensus.cc:729] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 3774a634d815476a80a6b57ac76edfd3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3774a634d815476a80a6b57ac76edfd3, State: Initialized, Role: FOLLOWER
I20260928 20:22:11.152747 18039 consensus_queue.cc:260] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 3774a634d815476a80a6b57ac76edfd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8cdfd447c5d4a09bdcc7cce530c7927" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 42799 } } peers { permanent_uuid: "3774a634d815476a80a6b57ac76edfd3" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34351 } } peers { permanent_uuid: "95fe79ad6abc4659bb584e646111be16" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 39927 } }
I20260928 20:22:11.153110 18039 ts_tablet_manager.cc:1428] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 3774a634d815476a80a6b57ac76edfd3: Time spent starting tablet: real 0.002s user 0.003s sys 0.002s
W20260928 20:22:11.240940 18151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:22:11.241180 18151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:22:11.241209 18151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:22:11.242522 18151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:22:11.242583 18151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:22:11.242614 18151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.2
I20260928 20:22:11.244844 18151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:45035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab
--principal=kudu/127.3.18.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.2:39927
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.18.2
--webserver_port=33175
--tserver_master_addrs=127.3.18.62:43799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.2
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:22:11.245163 18151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:22:11.245533 18151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:22:11.251626 18157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:22:11.251608 18158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:22:11.251871 18151 server_base.cc:1034] running on GCE node
W20260928 20:22:11.251653 18160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:22:11.252368 18151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:22:11.252774 18151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:22:11.253989 18151 hybrid_clock.cc:648] HybridClock initialized: now 1790626931253962 us; error 42 us; skew 500 ppm
Sep 28 20:22:11 dist-test-slave-b83v krb5kdc[16843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626931, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:22:11.257290 18151 init.cc:377] Logged in from keytab as kudu/127.3.18.2@KRBTEST.COM (short username kudu)
I20260928 20:22:11.257827 18151 webserver.cc:460] Webserver started at http://127.3.18.2:33175/ using document root <none> and password file <none>
I20260928 20:22:11.258167 18151 fs_manager.cc:359] Metadata directory not provided
I20260928 20:22:11.258249 18151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:22:11.259895 18151 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 20:22:11.260749 18167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:22:11.260987 18151 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:22:11.261121 18151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-1/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-1/wal
uuid: "95fe79ad6abc4659bb584e646111be16"
format_stamp: "Formatted at 2026-09-28 20:22:09 on dist-test-slave-b83v"
server_key: "76b44eaf5afd3f08644d29bbfdae8e5a"
server_key_iv: "1a009335852d99adead1cf73731429dc"
server_key_version: "encryptionkey@0"
I20260928 20:22:11.261549 18151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:22:11.292193 18151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:22:11.294831 18151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:22:11.295020 18151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:22:11.295276 18151 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:22:11.295674 18151 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:22:11.295732 18151 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:22:11.295790 18151 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:22:11.295820 18151 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:22:11.302402 18151 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.2:39927
I20260928 20:22:11.302434 18280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.2:39927 every 8 connection(s)
I20260928 20:22:11.302824 18151 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
I20260928 20:22:11.305351 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 18151
I20260928 20:22:11.305662 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.3:42799
--local_ip_for_outbound_sockets=127.3.18.3
--tserver_master_addrs=127.3.18.62:43799
--webserver_port=33697
--webserver_interface=127.3.18.3
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab
--principal=kudu/127.3.18.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:45035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:22:11.356554 18149 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:22:11.146492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:43799 (local address 127.3.18.1:39631)
0928 20:22:11.146719 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:11.146779 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 20:22:11.147492 (+ 713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:22:11.356327 (+208835us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:43799: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":89,"threads_started":1}
W20260928 20:22:11.356830 18147 heartbeater.cc:644] Failed to heartbeat to 127.3.18.62:43799 (0 consecutive failures): Network error: Failed to ping master at 127.3.18.62:43799: Client connection negotiation failed: client connection to 127.3.18.62:43799: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 20:22:11.357198 18283 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:22:11.303805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:43799 (local address 127.3.18.2:40727)
0928 20:22:11.303993 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:11.304035 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 20:22:11.304783 (+ 748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:22:11.356342 (+ 51559us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:43799: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":87,"threads_started":1}
I20260928 20:22:11.357197 18149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:11.357055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:43799 (local address 127.3.18.1:48745)
0928 20:22:11.357107 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:11.357167 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:43799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 20:22:11.357491 18281 heartbeater.cc:644] Failed to heartbeat to 127.3.18.62:43799 (0 consecutive failures): Network error: Failed to ping master at 127.3.18.62:43799: Client connection negotiation failed: client connection to 127.3.18.62:43799: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 20:22:11.357517 18149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:11.357431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:43799 (local address 127.3.18.1:57891)
0928 20:22:11.357468 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:11.357490 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:43799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260928 20:22:11.357627 18147 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 20:22:11.357817 18283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:11.357718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:43799 (local address 127.3.18.2:42735)
0928 20:22:11.357755 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:11.357798 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:43799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260928 20:22:11.358206 18283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:11.358063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:43799 (local address 127.3.18.2:43243)
0928 20:22:11.358141 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:11.358172 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:43799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260928 20:22:11.358349 18281 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 20:22:11.415607 18284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:22:11.415912 18284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:22:11.415966 18284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:22:11.417418 18284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:22:11.417483 18284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:22:11.417531 18284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.3
I20260928 20:22:11.419817 18284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:45035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab
--principal=kudu/127.3.18.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.3:42799
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.18.3
--webserver_port=33697
--tserver_master_addrs=127.3.18.62:43799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.3
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:22:11.420106 18284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:22:11.420357 18284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:22:11.428143 18290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:22:11.428282 18289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:22:11.428452 18284 server_base.cc:1034] running on GCE node
W20260928 20:22:11.428562 18292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:22:11.428794 18284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:22:11.429087 18284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:22:11.430239 18284 hybrid_clock.cc:648] HybridClock initialized: now 1790626931430214 us; error 47 us; skew 500 ppm
Sep 28 20:22:11 dist-test-slave-b83v krb5kdc[16843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626931, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:22:11.433655 18284 init.cc:377] Logged in from keytab as kudu/127.3.18.3@KRBTEST.COM (short username kudu)
I20260928 20:22:11.434170 18284 webserver.cc:460] Webserver started at http://127.3.18.3:33697/ using document root <none> and password file <none>
I20260928 20:22:11.434454 18284 fs_manager.cc:359] Metadata directory not provided
I20260928 20:22:11.434538 18284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:22:11.436254 18284 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 20:22:11.437049 18299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:22:11.437258 18284 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:22:11.437371 18284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-2/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-2/wal
uuid: "f8cdfd447c5d4a09bdcc7cce530c7927"
format_stamp: "Formatted at 2026-09-28 20:22:09 on dist-test-slave-b83v"
server_key: "00013873962e4a4a6f8c35699921c4c9"
server_key_iv: "631a85baaef20a20468698ebd11638a6"
server_key_version: "encryptionkey@0"
I20260928 20:22:11.437851 18284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:22:11.447844 18284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:22:11.450048 18284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:22:11.450232 18284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:22:11.450487 18284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:22:11.450816 18284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:22:11.450878 18284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:22:11.450923 18284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:22:11.450953 18284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:22:11.457432 18284 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.3:42799
I20260928 20:22:11.457487 18412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.3:42799 every 8 connection(s)
I20260928 20:22:11.457901 18284 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
I20260928 20:22:11.459121 18415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:11.458664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:43799 (local address 127.3.18.3:33651)
0928 20:22:11.458871 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:11.458947 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:43799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":82,"threads_started":1}
W20260928 20:22:11.459585 18413 heartbeater.cc:644] Failed to heartbeat to 127.3.18.62:43799 (0 consecutive failures): Network error: Failed to ping master at 127.3.18.62:43799: Client connection negotiation failed: client connection to 127.3.18.62:43799: connect: Connection refused (error 111)
I20260928 20:22:11.459985 18415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:11.459889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:43799 (local address 127.3.18.3:42061)
0928 20:22:11.459937 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:11.459965 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:43799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 20:22:11.460363 18415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:11.460288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:43799 (local address 127.3.18.3:60905)
0928 20:22:11.460329 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:11.460349 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:43799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 20:22:11.460492 18413 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 20:22:11.461810 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 18284
I20260928 20:22:12.358362 18417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:12.358073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:43799 (local address 127.3.18.1:50959)
0928 20:22:12.358244 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:12.358291 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:43799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":122,"threads_started":1}
I20260928 20:22:12.358953 18418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:12.358729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:43799 (local address 127.3.18.2:45969)
0928 20:22:12.358855 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:12.358895 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:43799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":59,"threads_started":1}
I20260928 20:22:12.447232 18419 raft_consensus.cc:482] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 3774a634d815476a80a6b57ac76edfd3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:22:12.447541 18419 raft_consensus.cc:504] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 3774a634d815476a80a6b57ac76edfd3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8cdfd447c5d4a09bdcc7cce530c7927" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 42799 } } peers { permanent_uuid: "3774a634d815476a80a6b57ac76edfd3" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34351 } } peers { permanent_uuid: "95fe79ad6abc4659bb584e646111be16" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 39927 } }
I20260928 20:22:12.447974 18419 leader_election.cc:288] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 3774a634d815476a80a6b57ac76edfd3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8cdfd447c5d4a09bdcc7cce530c7927 (127.3.18.3:42799), 95fe79ad6abc4659bb584e646111be16 (127.3.18.2:39927)
Sep 28 20:22:12 dist-test-slave-b83v krb5kdc[16843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626931, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.1@KRBTEST.COM for kudu/127.3.18.2@KRBTEST.COM
Sep 28 20:22:12 dist-test-slave-b83v krb5kdc[16843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626931, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.1@KRBTEST.COM for kudu/127.3.18.3@KRBTEST.COM
I20260928 20:22:12.461143 18423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:12.460902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:43799 (local address 127.3.18.3:38823)
0928 20:22:12.461044 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:12.461089 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:43799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":99,"threads_started":1}
I20260928 20:22:12.461544 18417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:12.448203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.3:42799 (local address 127.3.18.1:53167)
0928 20:22:12.448256 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:12.448269 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 20:22:12.448333 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:22:12.449543 (+ 1210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:22:12.449544 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:22:12.449697 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:22:12.449738 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:12.449740 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:12.451468 (+ 1728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:12.451469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:12.452910 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:12.452914 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:12.453961 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:12.453962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:12.454120 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:12.454892 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:22:12.454905 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:22:12.457586 (+ 2681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:22:12.459752 (+ 2166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:22:12.459789 (+ 37us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:22:12.459791 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:22:12.460102 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:22:12.460374 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:22:12.460375 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:22:12.460376 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:22:12.460492 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:22:12.460965 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:22:12.460972 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:22:12.461138 (+ 166us) client_negotiation.cc:769] Sending connection context
0928 20:22:12.461254 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 20:22:12.461447 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":92}
I20260928 20:22:12.461586 18421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:12.448399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:53167 (local address 127.3.18.3:42799)
0928 20:22:12.448588 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:12.448601 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:12.448628 (+ 27us) server_negotiation.cc:407] Connection header received
0928 20:22:12.449260 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:12.449263 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:12.449322 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:12.449503 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:22:12.449842 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:12.451291 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:12.453139 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:12.453815 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:12.453949 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:12.457707 (+ 3758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:22:12.457712 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:22:12.457712 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:22:12.457727 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:22:12.459658 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:12.460201 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:12.460204 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:12.460205 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:12.460299 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:12.460572 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:12.460573 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:12.460573 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:12.460886 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:22:12.460975 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:12.461346 (+ 371us) server_negotiation.cc:299] Negotiation successful
0928 20:22:12.461478 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":61,"threads_started":1}
I20260928 20:22:12.461748 18422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:12.448461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:51031 (local address 127.3.18.2:39927)
0928 20:22:12.448610 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:12.448614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:12.448626 (+ 12us) server_negotiation.cc:407] Connection header received
0928 20:22:12.448751 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:12.448754 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:12.448784 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:12.448988 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:22:12.449819 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:12.451297 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:12.453140 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:12.453816 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:12.453939 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:12.457225 (+ 3286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:22:12.457231 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:22:12.457232 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:22:12.457247 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:22:12.459667 (+ 2420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:12.460251 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:12.460256 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:12.460256 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:12.460355 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:12.460589 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:12.460590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:12.460591 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:12.460935 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:22:12.461399 (+ 464us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:12.461491 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 20:22:12.461627 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":68,"threads_started":1}
I20260928 20:22:12.461740 18420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:12.448232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.2:39927 (local address 127.3.18.1:51031)
0928 20:22:12.448348 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:12.448360 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:22:12.448441 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:22:12.449063 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:22:12.449065 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:22:12.449351 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:22:12.449649 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:12.449657 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:12.451447 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:12.451449 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:12.452910 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:12.452914 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:12.453960 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:12.453962 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:12.454120 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:12.454892 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:22:12.454905 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:22:12.457021 (+ 2116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:22:12.459802 (+ 2781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:22:12.459804 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:22:12.459805 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:22:12.460120 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:22:12.460437 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:22:12.460438 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:22:12.460438 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:22:12.460512 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:22:12.461037 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:22:12.461038 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:22:12.461136 (+ 98us) client_negotiation.cc:769] Sending connection context
0928 20:22:12.461239 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 20:22:12.461528 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"mutex_wait_us":203,"thread_start_us":62,"threads_started":1}
W20260928 20:22:12.463032 18035 leader_election.cc:341] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 3774a634d815476a80a6b57ac76edfd3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f8cdfd447c5d4a09bdcc7cce530c7927 (127.3.18.3:42799): Not found: Tablet not found: 246f5099c79d44cc9bbd11e6dbdcb9f4
W20260928 20:22:12.463158 18033 leader_election.cc:341] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 3774a634d815476a80a6b57ac76edfd3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 95fe79ad6abc4659bb584e646111be16 (127.3.18.2:39927): Not found: Tablet not found: 246f5099c79d44cc9bbd11e6dbdcb9f4
I20260928 20:22:12.463238 18033 leader_election.cc:302] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 3774a634d815476a80a6b57ac76edfd3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3774a634d815476a80a6b57ac76edfd3; no voters: 95fe79ad6abc4659bb584e646111be16, f8cdfd447c5d4a09bdcc7cce530c7927
I20260928 20:22:12.463339 18419 raft_consensus.cc:2726] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 3774a634d815476a80a6b57ac76edfd3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 20:22:13.359265 18424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:13.358985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:43799 (local address 127.3.18.1:42457)
0928 20:22:13.359148 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:13.359199 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:43799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":73,"threads_started":1}
I20260928 20:22:13.359702 18425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:13.359487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:43799 (local address 127.3.18.2:50815)
0928 20:22:13.359606 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:13.359641 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:43799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":56,"threads_started":1}
I20260928 20:22:13.461985 18426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:13.461741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:43799 (local address 127.3.18.3:34519)
0928 20:22:13.461871 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:13.461918 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:43799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":64,"threads_started":1}
I20260928 20:22:14.360260 18427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:14.360008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:43799 (local address 127.3.18.1:52369)
0928 20:22:14.360151 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:14.360206 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:43799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":93,"threads_started":1}
I20260928 20:22:14.360469 18428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:14.360232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:43799 (local address 127.3.18.2:35285)
0928 20:22:14.360349 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:14.360413 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:43799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":73,"threads_started":1}
I20260928 20:22:14.463003 18429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:14.462666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:43799 (local address 127.3.18.3:37457)
0928 20:22:14.462872 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:14.462935 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:43799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":75,"threads_started":1}
I20260928 20:22:14.567520 18430 raft_consensus.cc:482] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 3774a634d815476a80a6b57ac76edfd3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:22:14.567652 18430 raft_consensus.cc:504] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 3774a634d815476a80a6b57ac76edfd3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8cdfd447c5d4a09bdcc7cce530c7927" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 42799 } } peers { permanent_uuid: "3774a634d815476a80a6b57ac76edfd3" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34351 } } peers { permanent_uuid: "95fe79ad6abc4659bb584e646111be16" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 39927 } }
I20260928 20:22:14.567898 18430 leader_election.cc:288] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 3774a634d815476a80a6b57ac76edfd3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8cdfd447c5d4a09bdcc7cce530c7927 (127.3.18.3:42799), 95fe79ad6abc4659bb584e646111be16 (127.3.18.2:39927)
W20260928 20:22:14.568331 18033 leader_election.cc:341] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 3774a634d815476a80a6b57ac76edfd3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 95fe79ad6abc4659bb584e646111be16 (127.3.18.2:39927): Not found: Tablet not found: 246f5099c79d44cc9bbd11e6dbdcb9f4
W20260928 20:22:14.568403 18035 leader_election.cc:341] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 3774a634d815476a80a6b57ac76edfd3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f8cdfd447c5d4a09bdcc7cce530c7927 (127.3.18.3:42799): Not found: Tablet not found: 246f5099c79d44cc9bbd11e6dbdcb9f4
I20260928 20:22:14.568475 18035 leader_election.cc:302] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 3774a634d815476a80a6b57ac76edfd3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3774a634d815476a80a6b57ac76edfd3; no voters: 95fe79ad6abc4659bb584e646111be16, f8cdfd447c5d4a09bdcc7cce530c7927
I20260928 20:22:14.568575 18430 raft_consensus.cc:2726] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 3774a634d815476a80a6b57ac76edfd3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 20:22:15.360987 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:15.360715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:43799 (local address 127.3.18.1:54147)
0928 20:22:15.360876 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:15.360921 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:43799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":69,"threads_started":1}
I20260928 20:22:15.361182 18432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:15.360921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:43799 (local address 127.3.18.2:52621)
0928 20:22:15.361049 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:15.361133 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:43799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":70,"threads_started":1}
I20260928 20:22:15.464116 18433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:15.463601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:43799 (local address 127.3.18.3:34345)
0928 20:22:15.463737 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:15.464047 (+ 310us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:43799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":60,"threads_started":1}
I20260928 20:22:16.362003 18434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:16.361734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:43799 (local address 127.3.18.1:47521)
0928 20:22:16.361898 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:16.361952 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:43799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":76,"threads_started":1}
I20260928 20:22:16.362071 18435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:16.361780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:43799 (local address 127.3.18.2:36959)
0928 20:22:16.361977 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:16.362024 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:43799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":89,"threads_started":1}
I20260928 20:22:16.464946 18436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:16.464692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:43799 (local address 127.3.18.3:56751)
0928 20:22:16.464849 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:16.464896 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:43799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":109,"threads_started":1}
I20260928 20:22:16.481415 18437 raft_consensus.cc:482] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 3774a634d815476a80a6b57ac76edfd3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:22:16.481534 18437 raft_consensus.cc:504] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 3774a634d815476a80a6b57ac76edfd3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8cdfd447c5d4a09bdcc7cce530c7927" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 42799 } } peers { permanent_uuid: "3774a634d815476a80a6b57ac76edfd3" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34351 } } peers { permanent_uuid: "95fe79ad6abc4659bb584e646111be16" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 39927 } }
I20260928 20:22:16.481796 18437 leader_election.cc:288] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 3774a634d815476a80a6b57ac76edfd3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8cdfd447c5d4a09bdcc7cce530c7927 (127.3.18.3:42799), 95fe79ad6abc4659bb584e646111be16 (127.3.18.2:39927)
W20260928 20:22:16.482434 18035 leader_election.cc:341] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 3774a634d815476a80a6b57ac76edfd3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f8cdfd447c5d4a09bdcc7cce530c7927 (127.3.18.3:42799): Not found: Tablet not found: 246f5099c79d44cc9bbd11e6dbdcb9f4
W20260928 20:22:16.482646 18033 leader_election.cc:341] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 3774a634d815476a80a6b57ac76edfd3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 95fe79ad6abc4659bb584e646111be16 (127.3.18.2:39927): Not found: Tablet not found: 246f5099c79d44cc9bbd11e6dbdcb9f4
I20260928 20:22:16.482725 18033 leader_election.cc:302] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 3774a634d815476a80a6b57ac76edfd3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3774a634d815476a80a6b57ac76edfd3; no voters: 95fe79ad6abc4659bb584e646111be16, f8cdfd447c5d4a09bdcc7cce530c7927
I20260928 20:22:16.482802 18437 raft_consensus.cc:2726] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 3774a634d815476a80a6b57ac76edfd3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 20:22:17.362852 18439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:17.362598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:43799 (local address 127.3.18.2:34323)
0928 20:22:17.362780 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:17.362814 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:43799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":114,"threads_started":1}
I20260928 20:22:17.362848 18438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:17.362585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:43799 (local address 127.3.18.1:58031)
0928 20:22:17.362765 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:17.362800 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:43799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":73,"threads_started":1}
I20260928 20:22:17.466109 18440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:17.465766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:43799 (local address 127.3.18.3:48827)
0928 20:22:17.465978 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:17.466031 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:43799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":74,"threads_started":1}
I20260928 20:22:18.363667 18441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:18.363423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:43799 (local address 127.3.18.2:48541)
0928 20:22:18.363558 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:18.363602 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:43799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":84,"threads_started":1}
I20260928 20:22:18.364054 18442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:18.363728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:43799 (local address 127.3.18.1:49007)
0928 20:22:18.363935 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:18.363979 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:43799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":96,"threads_started":1}
I20260928 20:22:18.467029 18443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:18.466753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:43799 (local address 127.3.18.3:58157)
0928 20:22:18.466928 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:18.466978 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:43799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":70,"threads_started":1}
I20260928 20:22:19.364586 18444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:19.364291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:43799 (local address 127.3.18.2:45255)
0928 20:22:19.364461 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:19.364526 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:43799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":78,"threads_started":1}
I20260928 20:22:19.364842 18445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:19.364600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:43799 (local address 127.3.18.1:40331)
0928 20:22:19.364744 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:19.364789 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:43799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":64,"threads_started":1}
I20260928 20:22:19.467720 18446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:19.467498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:43799 (local address 127.3.18.3:52781)
0928 20:22:19.467635 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:19.467671 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:43799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":60,"threads_started":1}
I20260928 20:22:20.365509 18448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:20.365304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:43799 (local address 127.3.18.1:41095)
0928 20:22:20.365433 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:20.365467 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:43799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":74,"threads_started":1}
I20260928 20:22:20.365509 18447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:20.365260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:43799 (local address 127.3.18.2:58085)
0928 20:22:20.365405 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:20.365455 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:43799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":95,"threads_started":1}
I20260928 20:22:20.468678 18449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:20.468348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:43799 (local address 127.3.18.3:51371)
0928 20:22:20.468564 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:20.468624 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:43799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":77,"threads_started":1}
I20260928 20:22:20.925420 18450 raft_consensus.cc:482] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 3774a634d815476a80a6b57ac76edfd3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:22:20.925558 18450 raft_consensus.cc:504] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 3774a634d815476a80a6b57ac76edfd3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8cdfd447c5d4a09bdcc7cce530c7927" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 42799 } } peers { permanent_uuid: "3774a634d815476a80a6b57ac76edfd3" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34351 } } peers { permanent_uuid: "95fe79ad6abc4659bb584e646111be16" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 39927 } }
I20260928 20:22:20.925786 18450 leader_election.cc:288] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 3774a634d815476a80a6b57ac76edfd3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8cdfd447c5d4a09bdcc7cce530c7927 (127.3.18.3:42799), 95fe79ad6abc4659bb584e646111be16 (127.3.18.2:39927)
W20260928 20:22:20.926208 18033 leader_election.cc:341] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 3774a634d815476a80a6b57ac76edfd3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 95fe79ad6abc4659bb584e646111be16 (127.3.18.2:39927): Not found: Tablet not found: 246f5099c79d44cc9bbd11e6dbdcb9f4
W20260928 20:22:20.926319 18035 leader_election.cc:341] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 3774a634d815476a80a6b57ac76edfd3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f8cdfd447c5d4a09bdcc7cce530c7927 (127.3.18.3:42799): Not found: Tablet not found: 246f5099c79d44cc9bbd11e6dbdcb9f4
I20260928 20:22:20.926400 18035 leader_election.cc:302] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 3774a634d815476a80a6b57ac76edfd3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3774a634d815476a80a6b57ac76edfd3; no voters: 95fe79ad6abc4659bb584e646111be16, f8cdfd447c5d4a09bdcc7cce530c7927
I20260928 20:22:20.926501 18450 raft_consensus.cc:2726] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 3774a634d815476a80a6b57ac76edfd3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 20:22:21.366442 18452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:21.366213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:43799 (local address 127.3.18.1:56653)
0928 20:22:21.366353 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:21.366393 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:43799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":96,"threads_started":1}
I20260928 20:22:21.366443 18451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:21.366191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:43799 (local address 127.3.18.2:41953)
0928 20:22:21.366353 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:21.366403 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:43799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":63,"threads_started":1}
I20260928 20:22:21.469498 18453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:21.469261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:43799 (local address 127.3.18.3:42711)
0928 20:22:21.469390 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:21.469435 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:43799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":79,"threads_started":1}
I20260928 20:22:22.367298 18454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:22.367010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:43799 (local address 127.3.18.1:59865)
0928 20:22:22.367180 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:22.367237 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:43799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":74,"threads_started":1}
I20260928 20:22:22.367298 18455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:22.367009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:43799 (local address 127.3.18.2:37137)
0928 20:22:22.367214 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:22.367243 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:43799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":106,"threads_started":1}
I20260928 20:22:22.470494 18456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:22.470182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:43799 (local address 127.3.18.3:49581)
0928 20:22:22.470366 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:22.470409 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:43799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":99,"threads_started":1}
I20260928 20:22:23.368219 18457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:23.367975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:43799 (local address 127.3.18.1:50535)
0928 20:22:23.368111 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:23.368170 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:43799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":85,"threads_started":1}
I20260928 20:22:23.368218 18458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:23.367978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:43799 (local address 127.3.18.2:53119)
0928 20:22:23.368129 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:23.368172 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:43799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":83,"threads_started":1}
I20260928 20:22:23.471410 18459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:23.471142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:43799 (local address 127.3.18.3:54785)
0928 20:22:23.471306 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:23.471356 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:43799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":67,"threads_started":1}
I20260928 20:22:24.369030 18461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:24.368752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:43799 (local address 127.3.18.2:45669)
0928 20:22:24.368945 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:24.368977 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:43799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":103,"threads_started":1}
I20260928 20:22:24.369033 18460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:24.368758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:43799 (local address 127.3.18.1:52623)
0928 20:22:24.368915 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:24.368974 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:43799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":86,"threads_started":1}
I20260928 20:22:24.472339 18462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:24.472099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:43799 (local address 127.3.18.3:37955)
0928 20:22:24.472246 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:24.472287 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:43799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":60,"threads_started":1}
I20260928 20:22:25.369850 18463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:25.369596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:43799 (local address 127.3.18.1:47973)
0928 20:22:25.369761 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:25.369802 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:43799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":85,"threads_started":1}
I20260928 20:22:25.369858 18464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:25.369653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:43799 (local address 127.3.18.2:38229)
0928 20:22:25.369773 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:25.369806 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:43799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":48,"threads_started":1}
I20260928 20:22:25.473238 18465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:25.472967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:43799 (local address 127.3.18.3:56965)
0928 20:22:25.473142 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:25.473189 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:43799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":82,"threads_started":1}
I20260928 20:22:25.511277 18466 raft_consensus.cc:482] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 3774a634d815476a80a6b57ac76edfd3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:22:25.511387 18466 raft_consensus.cc:504] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 3774a634d815476a80a6b57ac76edfd3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8cdfd447c5d4a09bdcc7cce530c7927" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 42799 } } peers { permanent_uuid: "3774a634d815476a80a6b57ac76edfd3" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34351 } } peers { permanent_uuid: "95fe79ad6abc4659bb584e646111be16" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 39927 } }
I20260928 20:22:25.511684 18466 leader_election.cc:288] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 3774a634d815476a80a6b57ac76edfd3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8cdfd447c5d4a09bdcc7cce530c7927 (127.3.18.3:42799), 95fe79ad6abc4659bb584e646111be16 (127.3.18.2:39927)
W20260928 20:22:25.512166 18035 leader_election.cc:341] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 3774a634d815476a80a6b57ac76edfd3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f8cdfd447c5d4a09bdcc7cce530c7927 (127.3.18.3:42799): Not found: Tablet not found: 246f5099c79d44cc9bbd11e6dbdcb9f4
W20260928 20:22:25.512363 18033 leader_election.cc:341] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 3774a634d815476a80a6b57ac76edfd3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 95fe79ad6abc4659bb584e646111be16 (127.3.18.2:39927): Not found: Tablet not found: 246f5099c79d44cc9bbd11e6dbdcb9f4
I20260928 20:22:25.512436 18033 leader_election.cc:302] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 3774a634d815476a80a6b57ac76edfd3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3774a634d815476a80a6b57ac76edfd3; no voters: 95fe79ad6abc4659bb584e646111be16, f8cdfd447c5d4a09bdcc7cce530c7927
I20260928 20:22:25.512542 18466 raft_consensus.cc:2726] T 246f5099c79d44cc9bbd11e6dbdcb9f4 P 3774a634d815476a80a6b57ac76edfd3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 20:22:26.370863 18468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:26.370594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:43799 (local address 127.3.18.1:57291)
0928 20:22:26.370771 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:26.370815 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:43799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":84,"threads_started":1}
I20260928 20:22:26.370867 18467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:26.370594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:43799 (local address 127.3.18.2:35565)
0928 20:22:26.370792 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:26.370827 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:43799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":95,"threads_started":1}
I20260928 20:22:26.467808 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 17988
I20260928 20:22:26.473738 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 18151
I20260928 20:22:26.474910 18469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:26.473946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:43799 (local address 127.3.18.3:51331)
0928 20:22:26.474791 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:26.474848 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:43799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":821,"thread_start_us":59,"threads_started":1}
I20260928 20:22:26.479043 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 18284
2026-09-28T20:22:26Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18607 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:22:26 dist-test-slave-b83v krb5kdc[18472](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:22:26 dist-test-slave-b83v krb5kdc[18472](info): set up 2 sockets
Sep 28 20:22:26 dist-test-slave-b83v krb5kdc[18472](info): commencing operation
krb5kdc: starting...
W20260928 20:22:27.556512 3144 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Sep 28 20:22:27 dist-test-slave-b83v krb5kdc[18472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:22:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:22:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.18.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab.
Entry for principal kudu/127.3.18.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab.
I20260928 20:22:27.688761 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:40865
--webserver_interface=127.3.18.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:36695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.18.62:40865
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:22:27.773995 18485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:22:27.774240 18485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:22:27.774266 18485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:22:27.775465 18485 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 20:22:27.775506 18485 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 20:22:27.775518 18485 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 20:22:27.775530 18485 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:22:27.775544 18485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:22:27.775557 18485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:22:27.775570 18485 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:22:27.775583 18485 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:22:27.777189 18485 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:36695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.18.62:40865
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:40865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.18.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:22:27.777420 18485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:22:27.777612 18485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:22:27.784019 18490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:22:27.784104 18493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:22:27.784063 18485 server_base.cc:1034] running on GCE node
W20260928 20:22:27.784037 18491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:22:27.784677 18485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:22:27.785013 18485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:22:27.786203 18485 hybrid_clock.cc:648] HybridClock initialized: now 1790626947786202 us; error 55 us; skew 500 ppm
Sep 28 20:22:27 dist-test-slave-b83v krb5kdc[18472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626947, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:22:27.789191 18485 init.cc:377] Logged in from keytab as kudu/127.3.18.62@KRBTEST.COM (short username kudu)
I20260928 20:22:27.789669 18485 webserver.cc:460] Webserver started at http://127.3.18.62:45599/ using document root <none> and password file <none>
I20260928 20:22:27.789911 18485 fs_manager.cc:359] Metadata directory not provided
I20260928 20:22:27.789955 18485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:22:27.790053 18485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:22:27.791226 18485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/master-0/data/instance:
uuid: "fe5679c7c74f4b3e843599dc347a86bb"
format_stamp: "Formatted at 2026-09-28 20:22:27 on dist-test-slave-b83v"
server_key: "3c5671d9e419d5c04f18d0b2d10192a3"
server_key_iv: "dd70f2bec837b6baa8d55db353a06a97"
server_key_version: "encryptionkey@0"
I20260928 20:22:27.791565 18485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/master-0/wal/instance:
uuid: "fe5679c7c74f4b3e843599dc347a86bb"
format_stamp: "Formatted at 2026-09-28 20:22:27 on dist-test-slave-b83v"
server_key: "3c5671d9e419d5c04f18d0b2d10192a3"
server_key_iv: "dd70f2bec837b6baa8d55db353a06a97"
server_key_version: "encryptionkey@0"
I20260928 20:22:27.793088 18485 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:22:27.794096 18500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:22:27.794327 18485 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:22:27.794428 18485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/master-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/master-0/wal
uuid: "fe5679c7c74f4b3e843599dc347a86bb"
format_stamp: "Formatted at 2026-09-28 20:22:27 on dist-test-slave-b83v"
server_key: "3c5671d9e419d5c04f18d0b2d10192a3"
server_key_iv: "dd70f2bec837b6baa8d55db353a06a97"
server_key_version: "encryptionkey@0"
I20260928 20:22:27.794502 18485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:22:27.830410 18485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:22:27.832882 18485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:22:27.833017 18485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:22:27.837357 18485 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.62:40865
I20260928 20:22:27.837395 18552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.62:40865 every 8 connection(s)
I20260928 20:22:27.837781 18485 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
I20260928 20:22:27.838763 18553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:22:27.841472 18553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb: Bootstrap starting.
I20260928 20:22:27.842113 18553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb: Neither blocks nor log segments found. Creating new log.
I20260928 20:22:27.842437 18553 log.cc:824] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb: Log is configured to *not* fsync() on all Append() calls
I20260928 20:22:27.843372 18553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb: No bootstrap required, opened a new log
I20260928 20:22:27.844820 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 18485
I20260928 20:22:27.844914 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/master-0/wal/instance
I20260928 20:22:27.845156 3144 external_mini_cluster.cc:1428] Setting key 167c5bf3ce33ffea6532fa98fb2bb889
I20260928 20:22:27.845142 18553 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe5679c7c74f4b3e843599dc347a86bb" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 40865 } }
I20260928 20:22:27.845346 18553 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:22:27.845403 18553 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe5679c7c74f4b3e843599dc347a86bb, State: Initialized, Role: FOLLOWER
I20260928 20:22:27.845597 18553 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe5679c7c74f4b3e843599dc347a86bb" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 40865 } }
I20260928 20:22:27.845687 18553 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:22:27.845726 18553 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:22:27.845813 18553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:22:27.846947 18553 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe5679c7c74f4b3e843599dc347a86bb" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 40865 } }
I20260928 20:22:27.847083 18553 leader_election.cc:302] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe5679c7c74f4b3e843599dc347a86bb; no voters:
I20260928 20:22:27.847260 18553 leader_election.cc:288] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:22:27.847373 18558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:22:27.847522 18553 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:22:27.847616 18558 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb [term 1 LEADER]: Becoming Leader. State: Replica: fe5679c7c74f4b3e843599dc347a86bb, State: Running, Role: LEADER
I20260928 20:22:27.847791 18558 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe5679c7c74f4b3e843599dc347a86bb" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 40865 } }
I20260928 20:22:27.848244 18559 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe5679c7c74f4b3e843599dc347a86bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe5679c7c74f4b3e843599dc347a86bb" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 40865 } } }
I20260928 20:22:27.848349 18559 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb [sys.catalog]: This master's current role is: LEADER
I20260928 20:22:27.848534 18560 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe5679c7c74f4b3e843599dc347a86bb. Latest consensus state: current_term: 1 leader_uuid: "fe5679c7c74f4b3e843599dc347a86bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe5679c7c74f4b3e843599dc347a86bb" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 40865 } } }
I20260928 20:22:27.848592 18560 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb [sys.catalog]: This master's current role is: LEADER
I20260928 20:22:27.848693 18567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:22:27.849280 18567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:22:27.851285 18567 catalog_manager.cc:1347] Generated new cluster ID: a6286dddff7642f1866ae832f5093957
I20260928 20:22:27.851372 18567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 20:22:27 dist-test-slave-b83v krb5kdc[18472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:22:27.857987 18557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:27.845713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35408 (local address 127.3.18.62:40865)
0928 20:22:27.845923 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:27.845933 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:27.845969 (+ 36us) server_negotiation.cc:407] Connection header received
0928 20:22:27.846100 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:27.846108 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:27.846165 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:27.846330 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:22:27.847083 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:27.848799 (+ 1716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:27.849980 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:27.850467 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:27.850634 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:27.853635 (+ 3001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:22:27.853647 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:22:27.853648 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:22:27.853664 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:22:27.856006 (+ 2342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:27.856469 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:27.856473 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:27.856473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:27.856618 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:27.856851 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:27.856852 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:27.856852 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:27.857231 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:22:27.857397 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:27.857518 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 20:22:27.857747 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":103,"threads_started":1}
I20260928 20:22:27.868881 18567 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:22:27.869477 18567 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:22:27.876617 18567 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb: Generated new TSK 0
I20260928 20:22:27.877003 18567 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.18.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab.
Entry for principal kudu/127.3.18.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab.
I20260928 20:22:27.911746 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.1:0
--local_ip_for_outbound_sockets=127.3.18.1
--webserver_interface=127.3.18.1
--webserver_port=0
--tserver_master_addrs=127.3.18.62:40865
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab
--principal=kudu/127.3.18.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:36695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:22:27.999892 18579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:22:28.000159 18579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:22:28.000218 18579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:22:28.001523 18579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:22:28.001602 18579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:22:28.001652 18579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.1
I20260928 20:22:28.003235 18579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:36695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab
--principal=kudu/127.3.18.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.18.1
--webserver_port=0
--tserver_master_addrs=127.3.18.62:40865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.1
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:22:28.003525 18579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:22:28.003826 18579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:22:28.009267 18585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:22:28.009295 18584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:22:28.009272 18587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:22:28.009310 18579 server_base.cc:1034] running on GCE node
I20260928 20:22:28.009857 18579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:22:28.010129 18579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:22:28.011279 18579 hybrid_clock.cc:648] HybridClock initialized: now 1790626948011267 us; error 28 us; skew 500 ppm
Sep 28 20:22:28 dist-test-slave-b83v krb5kdc[18472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626948, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:22:28.014043 18579 init.cc:377] Logged in from keytab as kudu/127.3.18.1@KRBTEST.COM (short username kudu)
I20260928 20:22:28.014519 18579 webserver.cc:460] Webserver started at http://127.3.18.1:41063/ using document root <none> and password file <none>
I20260928 20:22:28.014721 18579 fs_manager.cc:359] Metadata directory not provided
I20260928 20:22:28.014762 18579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:22:28.014853 18579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:22:28.015971 18579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-0/data/instance:
uuid: "1073bb7be04e4025bd11239f3303b0e3"
format_stamp: "Formatted at 2026-09-28 20:22:28 on dist-test-slave-b83v"
server_key: "5f970228296b9f459911d0b49a1c54d3"
server_key_iv: "290dac0c00c973deaf9703386fdfbd18"
server_key_version: "encryptionkey@0"
I20260928 20:22:28.016280 18579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-0/wal/instance:
uuid: "1073bb7be04e4025bd11239f3303b0e3"
format_stamp: "Formatted at 2026-09-28 20:22:28 on dist-test-slave-b83v"
server_key: "5f970228296b9f459911d0b49a1c54d3"
server_key_iv: "290dac0c00c973deaf9703386fdfbd18"
server_key_version: "encryptionkey@0"
I20260928 20:22:28.017571 18579 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:22:28.018406 18594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:22:28.018602 18579 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:22:28.018685 18579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-0/wal
uuid: "1073bb7be04e4025bd11239f3303b0e3"
format_stamp: "Formatted at 2026-09-28 20:22:28 on dist-test-slave-b83v"
server_key: "5f970228296b9f459911d0b49a1c54d3"
server_key_iv: "290dac0c00c973deaf9703386fdfbd18"
server_key_version: "encryptionkey@0"
I20260928 20:22:28.018743 18579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:22:28.032174 18579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:22:28.034798 18579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:22:28.034925 18579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:22:28.035132 18579 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:22:28.035475 18579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:22:28.035513 18579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:22:28.035561 18579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:22:28.035589 18579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:22:28.041925 18579 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.1:34605
I20260928 20:22:28.041970 18707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.1:34605 every 8 connection(s)
I20260928 20:22:28.042317 18579 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
I20260928 20:22:28.047302 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 18579
I20260928 20:22:28.047400 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-0/wal/instance
I20260928 20:22:28.047845 3144 external_mini_cluster.cc:1428] Setting key 75bd28020341b56fb33bfa9eb0367ef9
Sep 28 20:22:28 dist-test-slave-b83v krb5kdc[18472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626948, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.1@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:22:28.053292 18557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:28.043086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:43109 (local address 127.3.18.62:40865)
0928 20:22:28.043135 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:28.043138 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:28.043365 (+ 227us) server_negotiation.cc:407] Connection header received
0928 20:22:28.043887 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:28.043889 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:28.043914 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:28.043971 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:22:28.045001 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:28.045524 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:28.046576 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:28.046797 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:28.046876 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:28.050022 (+ 3146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:22:28.050023 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:22:28.050024 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:22:28.050034 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:22:28.051659 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:28.052195 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:28.052196 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:28.052196 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:28.052261 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:28.052569 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:28.052569 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:28.052570 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:28.052692 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:22:28.052770 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:28.053076 (+ 306us) server_negotiation.cc:299] Negotiation successful
0928 20:22:28.053194 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 20:22:28.053498 18710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:28.043072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40865 (local address 127.3.18.1:43109)
0928 20:22:28.043225 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:28.043267 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 20:22:28.043750 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:22:28.044058 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:22:28.044063 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:22:28.044686 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:22:28.044886 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:28.044891 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:28.045628 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:28.045629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:28.046464 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:28.046467 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:28.046885 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:28.046886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:28.047077 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:28.047803 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:22:28.047816 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:22:28.049915 (+ 2099us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:22:28.051823 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:22:28.051833 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:22:28.051836 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:22:28.052091 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:22:28.052363 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:22:28.052365 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:22:28.052365 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:22:28.052492 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:22:28.052761 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:22:28.052768 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:22:28.052931 (+ 163us) client_negotiation.cc:769] Sending connection context
0928 20:22:28.053042 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 20:22:28.053227 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":83,"threads_started":1}
I20260928 20:22:28.054049 18708 heartbeater.cc:347] Connected to a master server at 127.3.18.62:40865
I20260928 20:22:28.054173 18708 heartbeater.cc:464] Registering TS with master...
I20260928 20:22:28.054401 18708 heartbeater.cc:511] Master 127.3.18.62:40865 requested a full tablet report, sending...
I20260928 20:22:28.054956 18517 ts_manager.cc:194] Registered new tserver with Master: 1073bb7be04e4025bd11239f3303b0e3 (127.3.18.1:34605)
I20260928 20:22:28.055605 18517 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.18.1@KRBTEST.COM'} at 127.3.18.1:43109
WARNING: no policy specified for kudu/127.3.18.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab.
Entry for principal kudu/127.3.18.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab.
I20260928 20:22:28.076933 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.2:0
--local_ip_for_outbound_sockets=127.3.18.2
--webserver_interface=127.3.18.2
--webserver_port=0
--tserver_master_addrs=127.3.18.62:40865
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab
--principal=kudu/127.3.18.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:36695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:22:28.163960 18713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:22:28.164211 18713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:22:28.164237 18713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:22:28.165520 18713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:22:28.165583 18713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:22:28.165613 18713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.2
I20260928 20:22:28.167191 18713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:36695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab
--principal=kudu/127.3.18.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.18.2
--webserver_port=0
--tserver_master_addrs=127.3.18.62:40865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.2
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:22:28.167433 18713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:22:28.167683 18713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:22:28.173535 18718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:22:28.173523 18719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:22:28.173586 18721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:22:28.174036 18713 server_base.cc:1034] running on GCE node
I20260928 20:22:28.174345 18713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:22:28.174641 18713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:22:28.175848 18713 hybrid_clock.cc:648] HybridClock initialized: now 1790626948175821 us; error 56 us; skew 500 ppm
Sep 28 20:22:28 dist-test-slave-b83v krb5kdc[18472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626948, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:22:28.178877 18713 init.cc:377] Logged in from keytab as kudu/127.3.18.2@KRBTEST.COM (short username kudu)
I20260928 20:22:28.179543 18713 webserver.cc:460] Webserver started at http://127.3.18.2:44901/ using document root <none> and password file <none>
I20260928 20:22:28.179868 18713 fs_manager.cc:359] Metadata directory not provided
I20260928 20:22:28.179937 18713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:22:28.180084 18713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:22:28.181250 18713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-1/data/instance:
uuid: "9891977fbea749aeb0f0e9e80b4f9361"
format_stamp: "Formatted at 2026-09-28 20:22:28 on dist-test-slave-b83v"
server_key: "0c19841604e3953e1aedad39d14244a9"
server_key_iv: "3300c18e4170e2c69334ff0566b0733a"
server_key_version: "encryptionkey@0"
I20260928 20:22:28.181658 18713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-1/wal/instance:
uuid: "9891977fbea749aeb0f0e9e80b4f9361"
format_stamp: "Formatted at 2026-09-28 20:22:28 on dist-test-slave-b83v"
server_key: "0c19841604e3953e1aedad39d14244a9"
server_key_iv: "3300c18e4170e2c69334ff0566b0733a"
server_key_version: "encryptionkey@0"
I20260928 20:22:28.183235 18713 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:22:28.184365 18728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:22:28.184712 18713 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 20:22:28.184886 18713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-1/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-1/wal
uuid: "9891977fbea749aeb0f0e9e80b4f9361"
format_stamp: "Formatted at 2026-09-28 20:22:28 on dist-test-slave-b83v"
server_key: "0c19841604e3953e1aedad39d14244a9"
server_key_iv: "3300c18e4170e2c69334ff0566b0733a"
server_key_version: "encryptionkey@0"
I20260928 20:22:28.184993 18713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:22:28.211645 18713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:22:28.214386 18713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:22:28.214687 18713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:22:28.214963 18713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:22:28.215302 18713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:22:28.215358 18713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:22:28.215401 18713 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:22:28.215431 18713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:22:28.222427 18713 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.2:43745
I20260928 20:22:28.222469 18841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.2:43745 every 8 connection(s)
I20260928 20:22:28.222886 18713 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
Sep 28 20:22:28 dist-test-slave-b83v krb5kdc[18472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626948, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.2@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:22:28.233215 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 18713
I20260928 20:22:28.233361 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-1/wal/instance
I20260928 20:22:28.233589 3144 external_mini_cluster.cc:1428] Setting key 2633ae3c2ec9bf1430c78713fb686e83
I20260928 20:22:28.233722 18557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:28.223689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:59521 (local address 127.3.18.62:40865)
0928 20:22:28.223756 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:28.223778 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:28.223980 (+ 202us) server_negotiation.cc:407] Connection header received
0928 20:22:28.224479 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:28.224481 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:28.224493 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:28.224562 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:22:28.225683 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:28.226233 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:28.227584 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:28.227819 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:28.227878 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:28.230477 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:22:28.230479 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:22:28.230479 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:22:28.230488 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:22:28.232114 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:28.232603 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:28.232605 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:28.232605 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:28.232670 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:28.232951 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:28.232952 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:28.232952 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:28.233092 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:22:28.233178 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:28.233506 (+ 328us) server_negotiation.cc:299] Negotiation successful
0928 20:22:28.233626 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 20:22:28.233984 18844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:28.223606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40865 (local address 127.3.18.2:59521)
0928 20:22:28.223805 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:28.223850 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 20:22:28.224355 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:22:28.224710 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:22:28.224714 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:22:28.225326 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:22:28.225546 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:28.225552 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:28.226546 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:28.226548 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:28.227470 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:28.227473 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:28.227987 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:28.227989 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:28.228115 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:28.228717 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:22:28.228725 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:22:28.230357 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:22:28.232237 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:22:28.232248 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:22:28.232251 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:22:28.232515 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:22:28.232744 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:22:28.232745 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:22:28.232746 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:22:28.232875 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:22:28.233170 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:22:28.233178 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:22:28.233340 (+ 162us) client_negotiation.cc:769] Sending connection context
0928 20:22:28.233469 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 20:22:28.233651 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":96,"threads_started":1}
I20260928 20:22:28.234669 18842 heartbeater.cc:347] Connected to a master server at 127.3.18.62:40865
I20260928 20:22:28.234787 18842 heartbeater.cc:464] Registering TS with master...
I20260928 20:22:28.235005 18842 heartbeater.cc:511] Master 127.3.18.62:40865 requested a full tablet report, sending...
I20260928 20:22:28.235452 18517 ts_manager.cc:194] Registered new tserver with Master: 9891977fbea749aeb0f0e9e80b4f9361 (127.3.18.2:43745)
I20260928 20:22:28.236030 18517 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.18.2@KRBTEST.COM'} at 127.3.18.2:59521
WARNING: no policy specified for kudu/127.3.18.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab.
Entry for principal kudu/127.3.18.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab.
I20260928 20:22:28.263650 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.3:0
--local_ip_for_outbound_sockets=127.3.18.3
--webserver_interface=127.3.18.3
--webserver_port=0
--tserver_master_addrs=127.3.18.62:40865
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab
--principal=kudu/127.3.18.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:36695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:22:28.350350 18847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:22:28.350574 18847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:22:28.350600 18847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:22:28.351879 18847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:22:28.351931 18847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:22:28.351958 18847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.3
I20260928 20:22:28.353483 18847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:36695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab
--principal=kudu/127.3.18.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.18.3
--webserver_port=0
--tserver_master_addrs=127.3.18.62:40865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.3
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:22:28.353691 18847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:22:28.353892 18847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:22:28.359270 18853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:22:28.359328 18855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:22:28.359311 18847 server_base.cc:1034] running on GCE node
W20260928 20:22:28.359263 18852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:22:28.359730 18847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:22:28.360056 18847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:22:28.361240 18847 hybrid_clock.cc:648] HybridClock initialized: now 1790626948361223 us; error 33 us; skew 500 ppm
Sep 28 20:22:28 dist-test-slave-b83v krb5kdc[18472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626948, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:22:28.364274 18847 init.cc:377] Logged in from keytab as kudu/127.3.18.3@KRBTEST.COM (short username kudu)
I20260928 20:22:28.364789 18847 webserver.cc:460] Webserver started at http://127.3.18.3:38473/ using document root <none> and password file <none>
I20260928 20:22:28.365020 18847 fs_manager.cc:359] Metadata directory not provided
I20260928 20:22:28.365067 18847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:22:28.365232 18847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:22:28.366354 18847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-2/data/instance:
uuid: "3638055a7dc94f378d5f16baf99f151d"
format_stamp: "Formatted at 2026-09-28 20:22:28 on dist-test-slave-b83v"
server_key: "03c355bf2c34de2ec07ddabb3293d980"
server_key_iv: "c64512be4d15bbebfff90112b37c8c6b"
server_key_version: "encryptionkey@0"
I20260928 20:22:28.366829 18847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-2/wal/instance:
uuid: "3638055a7dc94f378d5f16baf99f151d"
format_stamp: "Formatted at 2026-09-28 20:22:28 on dist-test-slave-b83v"
server_key: "03c355bf2c34de2ec07ddabb3293d980"
server_key_iv: "c64512be4d15bbebfff90112b37c8c6b"
server_key_version: "encryptionkey@0"
I20260928 20:22:28.368556 18847 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 20:22:28.369541 18862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:22:28.369728 18847 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:22:28.369831 18847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-2/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-2/wal
uuid: "3638055a7dc94f378d5f16baf99f151d"
format_stamp: "Formatted at 2026-09-28 20:22:28 on dist-test-slave-b83v"
server_key: "03c355bf2c34de2ec07ddabb3293d980"
server_key_iv: "c64512be4d15bbebfff90112b37c8c6b"
server_key_version: "encryptionkey@0"
I20260928 20:22:28.369895 18847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:22:28.383354 18847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:22:28.385635 18847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:22:28.385756 18847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:22:28.385988 18847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:22:28.386286 18847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:22:28.386317 18847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:22:28.386339 18847 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:22:28.386353 18847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:22:28.392822 18847 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.3:33365
I20260928 20:22:28.392863 18975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.3:33365 every 8 connection(s)
I20260928 20:22:28.393224 18847 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
I20260928 20:22:28.399705 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 18847
Sep 28 20:22:28 dist-test-slave-b83v krb5kdc[18472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626948, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.3@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:22:28.399857 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-2/wal/instance
I20260928 20:22:28.400076 3144 external_mini_cluster.cc:1428] Setting key 29e97f95061ef404ea57f09118b9f3aa
I20260928 20:22:28.403929 18557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:28.393975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:59061 (local address 127.3.18.62:40865)
0928 20:22:28.394037 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:28.394039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:28.394222 (+ 183us) server_negotiation.cc:407] Connection header received
0928 20:22:28.394773 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:28.394775 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:28.394788 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:28.394857 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:22:28.395957 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:28.396490 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:28.397559 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:28.397810 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:28.397878 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:28.400566 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:22:28.400567 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:22:28.400567 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:22:28.400574 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:22:28.402341 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:28.402865 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:28.402866 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:28.402866 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:28.402928 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:28.403234 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:28.403236 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:28.403236 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:28.403362 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:22:28.403438 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:28.403739 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 20:22:28.403839 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 20:22:28.404138 18978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:28.393935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40865 (local address 127.3.18.3:59061)
0928 20:22:28.394089 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:28.394142 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 20:22:28.394641 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:22:28.394940 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:22:28.394944 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:22:28.395581 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:22:28.395854 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:28.395860 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:28.396574 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:28.396575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:28.397473 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:28.397475 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:28.397911 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:28.397912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:28.398023 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:28.398550 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:22:28.398558 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:22:28.400420 (+ 1862us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:22:28.402452 (+ 2032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:22:28.402460 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:22:28.402462 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:22:28.402768 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:22:28.403017 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:22:28.403017 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:22:28.403018 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:22:28.403171 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:22:28.403430 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:22:28.403437 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:22:28.403580 (+ 143us) client_negotiation.cc:769] Sending connection context
0928 20:22:28.403692 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 20:22:28.403884 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":85,"threads_started":1}
I20260928 20:22:28.404654 18976 heartbeater.cc:347] Connected to a master server at 127.3.18.62:40865
I20260928 20:22:28.404781 18976 heartbeater.cc:464] Registering TS with master...
I20260928 20:22:28.404989 18976 heartbeater.cc:511] Master 127.3.18.62:40865 requested a full tablet report, sending...
I20260928 20:22:28.405352 18517 ts_manager.cc:194] Registered new tserver with Master: 3638055a7dc94f378d5f16baf99f151d (127.3.18.3:33365)
I20260928 20:22:28.405735 18517 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.18.3@KRBTEST.COM'} at 127.3.18.3:59061
I20260928 20:22:28.411890 3144 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 20:22:28.419437 18557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:28.413235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35412 (local address 127.3.18.62:40865)
0928 20:22:28.413289 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:28.413291 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:28.413402 (+ 111us) server_negotiation.cc:407] Connection header received
0928 20:22:28.413455 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:28.413456 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:28.413465 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:28.413520 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:22:28.414042 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:28.414526 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:28.415274 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:28.415456 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:28.415515 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:28.416866 (+ 1351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:22:28.416867 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:22:28.416867 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:22:28.416874 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:22:28.418425 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:28.418735 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:28.418736 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:28.418736 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:28.418788 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:28.418945 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:28.418945 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:28.418946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:28.419130 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:22:28.419211 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:28.419311 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 20:22:28.419371 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 20:22:28.421243 18517 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35412:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 20:22:28.423028 18517 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 20:22:28.430019 18988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:28.425761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.1:34605 (local address 127.0.0.1:47586)
0928 20:22:28.426142 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:28.426154 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:22:28.426237 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:22:28.426721 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:22:28.426722 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:22:28.426724 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:22:28.426777 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:28.426779 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:28.427707 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:28.427709 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:28.428821 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:28.428824 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:28.429685 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:28.429685 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:28.429777 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:28.429794 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 20:22:28.429873 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 20:22:28.429925 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":90,"threads_started":1}
I20260928 20:22:28.430042 18989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:28.425862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47586 (local address 127.3.18.1:34605)
0928 20:22:28.426029 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:28.426035 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:28.426334 (+ 299us) server_negotiation.cc:407] Connection header received
0928 20:22:28.426464 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:28.426466 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:28.426484 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:28.426556 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:22:28.426851 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:28.427591 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:28.428910 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:28.429544 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:28.429663 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:28.429732 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:28.429900 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 20:22:28.429949 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":85,"threads_started":1}
I20260928 20:22:28.430485 18987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:28.425752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.2:43745 (local address 127.0.0.1:57258)
0928 20:22:28.426065 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:28.426113 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 20:22:28.426260 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:22:28.426497 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:22:28.426501 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:22:28.426503 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:22:28.426592 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:28.426595 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:28.428048 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:28.428049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:28.429205 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:28.429207 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:28.430230 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:28.430231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:28.430295 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:28.430298 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 20:22:28.430368 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 20:22:28.430411 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":62,"threads_started":1}
I20260928 20:22:28.430502 18990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:28.425855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57258 (local address 127.3.18.2:43745)
0928 20:22:28.426075 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:28.426079 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:28.426230 (+ 151us) server_negotiation.cc:407] Connection header received
0928 20:22:28.426362 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:28.426364 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:28.426381 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:28.426482 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:22:28.426662 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:28.427928 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:28.429292 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:28.430039 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:28.430115 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:28.430193 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:28.430385 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 20:22:28.430427 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":106,"threads_started":1}
I20260928 20:22:28.430687 18991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:28.425974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.3:33365 (local address 127.0.0.1:58176)
0928 20:22:28.426183 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:28.426192 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 20:22:28.426237 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:22:28.427149 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:22:28.427150 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:22:28.427152 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:22:28.427191 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:28.427192 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:28.428503 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:28.428504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:28.429491 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:28.429492 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:28.430458 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:28.430459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:28.430520 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:28.430523 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 20:22:28.430569 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 20:22:28.430610 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":68,"threads_started":1}
I20260928 20:22:28.430912 18992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:28.426192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58176 (local address 127.3.18.3:33365)
0928 20:22:28.426927 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:28.426932 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:28.426940 (+ 8us) server_negotiation.cc:407] Connection header received
0928 20:22:28.426978 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:28.426980 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:28.426995 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:28.427068 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:22:28.427269 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:28.428393 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:28.429582 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:28.430304 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:28.430366 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:28.430425 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:28.430795 (+ 370us) server_negotiation.cc:299] Negotiation successful
0928 20:22:28.430836 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":697,"thread_start_us":115,"threads_started":1}
I20260928 20:22:28.431324 18642 tablet_service.cc:1467] Processing CreateTablet for tablet 6d8762e30b5c4b8398cb3f0fcc075ab5 (DEFAULT_TABLE table=test-table [id=93e1921d6ef94edb9b7e3ab1fcfd7754]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:22:28.431402 18776 tablet_service.cc:1467] Processing CreateTablet for tablet 6d8762e30b5c4b8398cb3f0fcc075ab5 (DEFAULT_TABLE table=test-table [id=93e1921d6ef94edb9b7e3ab1fcfd7754]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:22:28.431560 18910 tablet_service.cc:1467] Processing CreateTablet for tablet 6d8762e30b5c4b8398cb3f0fcc075ab5 (DEFAULT_TABLE table=test-table [id=93e1921d6ef94edb9b7e3ab1fcfd7754]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:22:28.431725 18642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d8762e30b5c4b8398cb3f0fcc075ab5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:22:28.431794 18776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d8762e30b5c4b8398cb3f0fcc075ab5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:22:28.431895 18910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d8762e30b5c4b8398cb3f0fcc075ab5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:22:28.434710 18993 tablet_bootstrap.cc:492] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 1073bb7be04e4025bd11239f3303b0e3: Bootstrap starting.
I20260928 20:22:28.435135 18994 tablet_bootstrap.cc:492] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 9891977fbea749aeb0f0e9e80b4f9361: Bootstrap starting.
I20260928 20:22:28.435477 18993 tablet_bootstrap.cc:654] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 1073bb7be04e4025bd11239f3303b0e3: Neither blocks nor log segments found. Creating new log.
I20260928 20:22:28.435912 18994 tablet_bootstrap.cc:654] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 9891977fbea749aeb0f0e9e80b4f9361: Neither blocks nor log segments found. Creating new log.
I20260928 20:22:28.435989 18995 tablet_bootstrap.cc:492] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 3638055a7dc94f378d5f16baf99f151d: Bootstrap starting.
I20260928 20:22:28.436160 18993 log.cc:824] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 1073bb7be04e4025bd11239f3303b0e3: Log is configured to *not* fsync() on all Append() calls
I20260928 20:22:28.436503 18994 log.cc:824] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 9891977fbea749aeb0f0e9e80b4f9361: Log is configured to *not* fsync() on all Append() calls
I20260928 20:22:28.436707 18995 tablet_bootstrap.cc:654] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 3638055a7dc94f378d5f16baf99f151d: Neither blocks nor log segments found. Creating new log.
I20260928 20:22:28.437007 18993 tablet_bootstrap.cc:492] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 1073bb7be04e4025bd11239f3303b0e3: No bootstrap required, opened a new log
I20260928 20:22:28.437086 18993 ts_tablet_manager.cc:1397] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 1073bb7be04e4025bd11239f3303b0e3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 20:22:28.437280 18994 tablet_bootstrap.cc:492] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 9891977fbea749aeb0f0e9e80b4f9361: No bootstrap required, opened a new log
I20260928 20:22:28.437314 18995 log.cc:824] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 3638055a7dc94f378d5f16baf99f151d: Log is configured to *not* fsync() on all Append() calls
I20260928 20:22:28.437441 18994 ts_tablet_manager.cc:1397] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 9891977fbea749aeb0f0e9e80b4f9361: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 20:22:28.439242 18993 raft_consensus.cc:348] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 1073bb7be04e4025bd11239f3303b0e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891977fbea749aeb0f0e9e80b4f9361" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 43745 } } peers { permanent_uuid: "3638055a7dc94f378d5f16baf99f151d" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 33365 } } peers { permanent_uuid: "1073bb7be04e4025bd11239f3303b0e3" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34605 } }
I20260928 20:22:28.439443 18993 raft_consensus.cc:374] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 1073bb7be04e4025bd11239f3303b0e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:22:28.439481 18993 raft_consensus.cc:729] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 1073bb7be04e4025bd11239f3303b0e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1073bb7be04e4025bd11239f3303b0e3, State: Initialized, Role: FOLLOWER
I20260928 20:22:28.439421 18994 raft_consensus.cc:348] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 9891977fbea749aeb0f0e9e80b4f9361 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891977fbea749aeb0f0e9e80b4f9361" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 43745 } } peers { permanent_uuid: "3638055a7dc94f378d5f16baf99f151d" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 33365 } } peers { permanent_uuid: "1073bb7be04e4025bd11239f3303b0e3" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34605 } }
I20260928 20:22:28.439589 18994 raft_consensus.cc:374] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 9891977fbea749aeb0f0e9e80b4f9361 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:22:28.439642 18994 raft_consensus.cc:729] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 9891977fbea749aeb0f0e9e80b4f9361 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9891977fbea749aeb0f0e9e80b4f9361, State: Initialized, Role: FOLLOWER
I20260928 20:22:28.439615 18993 consensus_queue.cc:260] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 1073bb7be04e4025bd11239f3303b0e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891977fbea749aeb0f0e9e80b4f9361" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 43745 } } peers { permanent_uuid: "3638055a7dc94f378d5f16baf99f151d" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 33365 } } peers { permanent_uuid: "1073bb7be04e4025bd11239f3303b0e3" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34605 } }
I20260928 20:22:28.439750 18994 consensus_queue.cc:260] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 9891977fbea749aeb0f0e9e80b4f9361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891977fbea749aeb0f0e9e80b4f9361" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 43745 } } peers { permanent_uuid: "3638055a7dc94f378d5f16baf99f151d" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 33365 } } peers { permanent_uuid: "1073bb7be04e4025bd11239f3303b0e3" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34605 } }
I20260928 20:22:28.439944 18708 heartbeater.cc:503] Master 127.3.18.62:40865 was elected leader, sending a full tablet report...
I20260928 20:22:28.440107 18994 ts_tablet_manager.cc:1428] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 9891977fbea749aeb0f0e9e80b4f9361: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260928 20:22:28.439941 18993 ts_tablet_manager.cc:1428] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 1073bb7be04e4025bd11239f3303b0e3: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260928 20:22:28.440361 18995 tablet_bootstrap.cc:492] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 3638055a7dc94f378d5f16baf99f151d: No bootstrap required, opened a new log
I20260928 20:22:28.440371 18842 heartbeater.cc:503] Master 127.3.18.62:40865 was elected leader, sending a full tablet report...
I20260928 20:22:28.440435 18995 ts_tablet_manager.cc:1397] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 3638055a7dc94f378d5f16baf99f151d: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260928 20:22:28.442231 18995 raft_consensus.cc:348] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 3638055a7dc94f378d5f16baf99f151d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891977fbea749aeb0f0e9e80b4f9361" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 43745 } } peers { permanent_uuid: "3638055a7dc94f378d5f16baf99f151d" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 33365 } } peers { permanent_uuid: "1073bb7be04e4025bd11239f3303b0e3" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34605 } }
I20260928 20:22:28.442354 18995 raft_consensus.cc:374] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 3638055a7dc94f378d5f16baf99f151d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:22:28.442374 18995 raft_consensus.cc:729] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 3638055a7dc94f378d5f16baf99f151d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3638055a7dc94f378d5f16baf99f151d, State: Initialized, Role: FOLLOWER
I20260928 20:22:28.442484 18995 consensus_queue.cc:260] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 3638055a7dc94f378d5f16baf99f151d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891977fbea749aeb0f0e9e80b4f9361" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 43745 } } peers { permanent_uuid: "3638055a7dc94f378d5f16baf99f151d" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 33365 } } peers { permanent_uuid: "1073bb7be04e4025bd11239f3303b0e3" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34605 } }
I20260928 20:22:28.442716 18995 ts_tablet_manager.cc:1428] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 3638055a7dc94f378d5f16baf99f151d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 20:22:28.442833 18976 heartbeater.cc:503] Master 127.3.18.62:40865 was elected leader, sending a full tablet report...
I20260928 20:22:28.464064 19000 raft_consensus.cc:482] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 9891977fbea749aeb0f0e9e80b4f9361 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:22:28.464188 19000 raft_consensus.cc:504] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 9891977fbea749aeb0f0e9e80b4f9361 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891977fbea749aeb0f0e9e80b4f9361" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 43745 } } peers { permanent_uuid: "3638055a7dc94f378d5f16baf99f151d" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 33365 } } peers { permanent_uuid: "1073bb7be04e4025bd11239f3303b0e3" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34605 } }
I20260928 20:22:28.464674 19000 leader_election.cc:288] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 9891977fbea749aeb0f0e9e80b4f9361 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3638055a7dc94f378d5f16baf99f151d (127.3.18.3:33365), 1073bb7be04e4025bd11239f3303b0e3 (127.3.18.1:34605)
I20260928 20:22:28.468827 18992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:28.464936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:49859 (local address 127.3.18.3:33365)
0928 20:22:28.464998 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:28.465001 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:28.465010 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:22:28.465049 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:28.465050 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:28.465066 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:28.465137 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:22:28.465337 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:28.466146 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:28.467734 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:28.468362 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:28.468448 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:28.468460 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:28.468705 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 20:22:28.468752 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 20:22:28.468827 18989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:28.464965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:58253 (local address 127.3.18.1:34605)
0928 20:22:28.465011 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:28.465013 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:28.465099 (+ 86us) server_negotiation.cc:407] Connection header received
0928 20:22:28.465168 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:28.465170 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:28.465181 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:28.465233 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:22:28.465399 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:28.466140 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:28.467734 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:28.468358 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:28.468448 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:28.468460 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:28.468712 (+ 252us) server_negotiation.cc:299] Negotiation successful
0928 20:22:28.468754 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 20:22:28.468838 18844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:28.464771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.3:33365 (local address 127.3.18.2:49859)
0928 20:22:28.464828 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:28.464840 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:22:28.464883 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:22:28.465166 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:22:28.465167 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:22:28.465169 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:22:28.465267 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:28.465269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:28.466242 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:28.466243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:28.467647 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:28.467648 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:28.468502 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:28.468503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:28.468601 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:28.468603 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 20:22:28.468672 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 20:22:28.468731 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":67}
I20260928 20:22:28.468845 19002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:28.464837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.1:34605 (local address 127.3.18.2:58253)
0928 20:22:28.465024 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:28.465033 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 20:22:28.465094 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:22:28.465257 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:22:28.465257 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:22:28.465259 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:22:28.465330 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:28.465331 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:28.466242 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:28.466243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:28.467624 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:28.467628 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:28.468503 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:28.468504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:28.468595 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:28.468600 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 20:22:28.468706 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 20:22:28.468739 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":81,"thread_start_us":121,"threads_started":1}
I20260928 20:22:28.469230 18662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d8762e30b5c4b8398cb3f0fcc075ab5" candidate_uuid: "9891977fbea749aeb0f0e9e80b4f9361" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1073bb7be04e4025bd11239f3303b0e3" is_pre_election: true
I20260928 20:22:28.469241 18930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d8762e30b5c4b8398cb3f0fcc075ab5" candidate_uuid: "9891977fbea749aeb0f0e9e80b4f9361" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3638055a7dc94f378d5f16baf99f151d" is_pre_election: true
I20260928 20:22:28.469419 18930 raft_consensus.cc:2454] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 3638055a7dc94f378d5f16baf99f151d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9891977fbea749aeb0f0e9e80b4f9361 in term 0.
I20260928 20:22:28.469427 18662 raft_consensus.cc:2454] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 1073bb7be04e4025bd11239f3303b0e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9891977fbea749aeb0f0e9e80b4f9361 in term 0.
I20260928 20:22:28.469733 18730 leader_election.cc:302] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 9891977fbea749aeb0f0e9e80b4f9361 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1073bb7be04e4025bd11239f3303b0e3, 9891977fbea749aeb0f0e9e80b4f9361; no voters:
I20260928 20:22:28.469889 19000 raft_consensus.cc:2781] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 9891977fbea749aeb0f0e9e80b4f9361 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 20:22:28.469954 19000 raft_consensus.cc:482] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 9891977fbea749aeb0f0e9e80b4f9361 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 20:22:28.469986 19000 raft_consensus.cc:3037] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 9891977fbea749aeb0f0e9e80b4f9361 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:22:28.470758 19000 raft_consensus.cc:504] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 9891977fbea749aeb0f0e9e80b4f9361 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891977fbea749aeb0f0e9e80b4f9361" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 43745 } } peers { permanent_uuid: "3638055a7dc94f378d5f16baf99f151d" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 33365 } } peers { permanent_uuid: "1073bb7be04e4025bd11239f3303b0e3" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34605 } }
I20260928 20:22:28.470937 19000 leader_election.cc:288] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 9891977fbea749aeb0f0e9e80b4f9361 [CANDIDATE]: Term 1 election: Requested vote from peers 3638055a7dc94f378d5f16baf99f151d (127.3.18.3:33365), 1073bb7be04e4025bd11239f3303b0e3 (127.3.18.1:34605)
I20260928 20:22:28.471145 18662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d8762e30b5c4b8398cb3f0fcc075ab5" candidate_uuid: "9891977fbea749aeb0f0e9e80b4f9361" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1073bb7be04e4025bd11239f3303b0e3"
I20260928 20:22:28.471179 18930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d8762e30b5c4b8398cb3f0fcc075ab5" candidate_uuid: "9891977fbea749aeb0f0e9e80b4f9361" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3638055a7dc94f378d5f16baf99f151d"
I20260928 20:22:28.471254 18662 raft_consensus.cc:3037] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 1073bb7be04e4025bd11239f3303b0e3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:22:28.471273 18930 raft_consensus.cc:3037] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 3638055a7dc94f378d5f16baf99f151d [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:22:28.472180 18930 raft_consensus.cc:2454] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 3638055a7dc94f378d5f16baf99f151d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9891977fbea749aeb0f0e9e80b4f9361 in term 1.
I20260928 20:22:28.472198 18662 raft_consensus.cc:2454] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 1073bb7be04e4025bd11239f3303b0e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9891977fbea749aeb0f0e9e80b4f9361 in term 1.
I20260928 20:22:28.472378 18729 leader_election.cc:302] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 9891977fbea749aeb0f0e9e80b4f9361 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3638055a7dc94f378d5f16baf99f151d, 9891977fbea749aeb0f0e9e80b4f9361; no voters:
I20260928 20:22:28.472561 19000 raft_consensus.cc:2781] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 9891977fbea749aeb0f0e9e80b4f9361 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:22:28.472728 19000 raft_consensus.cc:686] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 9891977fbea749aeb0f0e9e80b4f9361 [term 1 LEADER]: Becoming Leader. State: Replica: 9891977fbea749aeb0f0e9e80b4f9361, State: Running, Role: LEADER
I20260928 20:22:28.472834 19000 consensus_queue.cc:237] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 9891977fbea749aeb0f0e9e80b4f9361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891977fbea749aeb0f0e9e80b4f9361" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 43745 } } peers { permanent_uuid: "3638055a7dc94f378d5f16baf99f151d" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 33365 } } peers { permanent_uuid: "1073bb7be04e4025bd11239f3303b0e3" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34605 } }
W20260928 20:22:28.473461 18843 tablet.cc:2369] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 9891977fbea749aeb0f0e9e80b4f9361: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 20:22:28.473714 18517 catalog_manager.cc:5495] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 9891977fbea749aeb0f0e9e80b4f9361 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9891977fbea749aeb0f0e9e80b4f9361 (127.3.18.2). New cstate: current_term: 1 leader_uuid: "9891977fbea749aeb0f0e9e80b4f9361" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891977fbea749aeb0f0e9e80b4f9361" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 43745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3638055a7dc94f378d5f16baf99f151d" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 33365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1073bb7be04e4025bd11239f3303b0e3" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34605 } health_report { overall_health: UNKNOWN } } }
I20260928 20:22:28.488884 18990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:28.486180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57266 (local address 127.3.18.2:43745)
0928 20:22:28.486340 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:28.486342 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:28.486356 (+ 14us) server_negotiation.cc:407] Connection header received
0928 20:22:28.486392 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:28.486393 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:28.486415 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:28.486490 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:22:28.486650 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:28.487227 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:28.488106 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:28.488325 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:28.488397 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:28.488517 (+ 120us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:22:28.488656 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:22:28.488740 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:28.488759 (+ 19us) server_negotiation.cc:299] Negotiation successful
0928 20:22:28.488777 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136}
I20260928 20:22:28.491036 18930 raft_consensus.cc:1261] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 3638055a7dc94f378d5f16baf99f151d [term 1 FOLLOWER]: Refusing update from remote peer 9891977fbea749aeb0f0e9e80b4f9361: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:22:28.491070 18662 raft_consensus.cc:1261] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 1073bb7be04e4025bd11239f3303b0e3 [term 1 FOLLOWER]: Refusing update from remote peer 9891977fbea749aeb0f0e9e80b4f9361: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:22:28.491398 19000 consensus_queue.cc:1035] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 9891977fbea749aeb0f0e9e80b4f9361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3638055a7dc94f378d5f16baf99f151d" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 33365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:22:28.491564 19003 consensus_queue.cc:1035] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 9891977fbea749aeb0f0e9e80b4f9361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1073bb7be04e4025bd11239f3303b0e3" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:22:28.494103 19006 mvcc.cc:205] Tried to move back new op lower bound from 7334407981015670784 to 7334407980945338368. Current Snapshot: MvccSnapshot[applied={T|T < 7334407981015670784}]
I20260928 20:22:28.494715 19008 mvcc.cc:205] Tried to move back new op lower bound from 7334407981015670784 to 7334407980945338368. Current Snapshot: MvccSnapshot[applied={T|T < 7334407981015670784}]
I20260928 20:22:28.496634 18517 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:22:28.496824 18517 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:22:28.497907 18517 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb: Sending DeleteTablet for 3 replicas of tablet 6d8762e30b5c4b8398cb3f0fcc075ab5
I20260928 20:22:28.498287 18776 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d8762e30b5c4b8398cb3f0fcc075ab5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:22:28 UTC) from {username='kudu', principal='kudu/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:57258
I20260928 20:22:28.498399 18910 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d8762e30b5c4b8398cb3f0fcc075ab5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:22:28 UTC) from {username='kudu', principal='kudu/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:58176
I20260928 20:22:28.498868 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 18579
W20260928 20:22:28.505782 18502 connection.cc:535] client connection to 127.3.18.1:34605 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 20:22:28.505918 18502 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 20:22:28.505930 19017 tablet_replica.cc:330] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 3638055a7dc94f378d5f16baf99f151d: stopping tablet replica
I20260928 20:22:28.506072 19017 raft_consensus.cc:2229] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 3638055a7dc94f378d5f16baf99f151d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:22:28.506199 19017 raft_consensus.cc:2258] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 3638055a7dc94f378d5f16baf99f151d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:22:28.506417 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 18713
I20260928 20:22:28.506523 18991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:28.506378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.1:34605 (local address 127.0.0.1:47594)
0928 20:22:28.506438 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:28.506480 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.1:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 20:22:28.507356 19017 ts_tablet_manager.cc:1905] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 3638055a7dc94f378d5f16baf99f151d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 20:22:28.507504 18502 catalog_manager.cc:4584] TS 1073bb7be04e4025bd11239f3303b0e3 (127.3.18.1:34605): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6d8762e30b5c4b8398cb3f0fcc075ab5: Network error: Client connection negotiation failed: client connection to 127.3.18.1:34605: connect: Connection refused (error 111)
I20260928 20:22:28.509397 19017 ts_tablet_manager.cc:1918] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 3638055a7dc94f378d5f16baf99f151d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:22:28.509585 19017 log.cc:1196] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 3638055a7dc94f378d5f16baf99f151d: Deleting WAL directory at /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-2/wal/wals/6d8762e30b5c4b8398cb3f0fcc075ab5
I20260928 20:22:28.509891 19017 ts_tablet_manager.cc:1939] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 3638055a7dc94f378d5f16baf99f151d: Deleting consensus metadata
I20260928 20:22:28.510401 18501 catalog_manager.cc:4841] TS 3638055a7dc94f378d5f16baf99f151d (127.3.18.3:33365): tablet 6d8762e30b5c4b8398cb3f0fcc075ab5 (table test-table [id=93e1921d6ef94edb9b7e3ab1fcfd7754]) successfully deleted
I20260928 20:22:28.512661 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 18847
I20260928 20:22:28.512876 18991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:28.512744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.2:43745 (local address 127.0.0.1:57270)
0928 20:22:28.512797 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:28.512830 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.2:43745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260928 20:22:28.517746 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 18485
I20260928 20:22:48.523671 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:40865
--webserver_interface=127.3.18.62
--webserver_port=45599
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:36695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.18.62:40865
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:22:48.612679 19019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:22:48.612897 19019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:22:48.612923 19019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:22:48.614282 19019 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 20:22:48.614336 19019 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 20:22:48.614349 19019 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 20:22:48.614362 19019 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:22:48.614377 19019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:22:48.614389 19019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:22:48.614401 19019 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:22:48.614456 19019 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:22:48.616087 19019 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:36695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.18.62:40865
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:40865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.18.62
--webserver_port=45599
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:22:48.616387 19019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:22:48.616644 19019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:22:48.626345 19028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:22:48.626546 19025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:22:48.626379 19019 server_base.cc:1034] running on GCE node
W20260928 20:22:48.626705 19026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:22:48.627079 19019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:22:48.627487 19019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:22:48.628670 19019 hybrid_clock.cc:648] HybridClock initialized: now 1790626968628659 us; error 30 us; skew 500 ppm
Sep 28 20:22:48 dist-test-slave-b83v krb5kdc[18472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626968, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:22:48.638157 19019 init.cc:377] Logged in from keytab as kudu/127.3.18.62@KRBTEST.COM (short username kudu)
I20260928 20:22:48.638610 19019 webserver.cc:460] Webserver started at http://127.3.18.62:45599/ using document root <none> and password file <none>
I20260928 20:22:48.638892 19019 fs_manager.cc:359] Metadata directory not provided
I20260928 20:22:48.638964 19019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:22:48.640563 19019 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 20:22:48.641312 19035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:22:48.641549 19019 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:22:48.641721 19019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/master-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/master-0/wal
uuid: "fe5679c7c74f4b3e843599dc347a86bb"
format_stamp: "Formatted at 2026-09-28 20:22:27 on dist-test-slave-b83v"
server_key: "3c5671d9e419d5c04f18d0b2d10192a3"
server_key_iv: "dd70f2bec837b6baa8d55db353a06a97"
server_key_version: "encryptionkey@0"
I20260928 20:22:48.642151 19019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:22:48.652191 19019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:22:48.655083 19019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:22:48.655290 19019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:22:48.659649 19019 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.62:40865
I20260928 20:22:48.659694 19087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.62:40865 every 8 connection(s)
I20260928 20:22:48.660127 19019 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
I20260928 20:22:48.661669 19088 sys_catalog.cc:263] Verifying existing consensus state
I20260928 20:22:48.662324 19088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb: Bootstrap starting.
I20260928 20:22:48.669719 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 19019
I20260928 20:22:48.670089 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.1:34605
--local_ip_for_outbound_sockets=127.3.18.1
--tserver_master_addrs=127.3.18.62:40865
--webserver_port=41063
--webserver_interface=127.3.18.1
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab
--principal=kudu/127.3.18.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:36695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 20:22:48.671406 19088 log.cc:824] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb: Log is configured to *not* fsync() on all Append() calls
I20260928 20:22:48.673739 19088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 20:22:48.674103 19088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb: Bootstrap complete.
I20260928 20:22:48.676530 19088 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe5679c7c74f4b3e843599dc347a86bb" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 40865 } }
I20260928 20:22:48.676787 19088 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe5679c7c74f4b3e843599dc347a86bb, State: Initialized, Role: FOLLOWER
I20260928 20:22:48.676999 19088 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe5679c7c74f4b3e843599dc347a86bb" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 40865 } }
I20260928 20:22:48.677106 19088 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:22:48.677166 19088 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:22:48.677218 19088 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb [term 1 FOLLOWER]: Advancing to term 2
I20260928 20:22:48.678232 19088 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe5679c7c74f4b3e843599dc347a86bb" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 40865 } }
I20260928 20:22:48.678351 19088 leader_election.cc:302] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe5679c7c74f4b3e843599dc347a86bb; no voters:
I20260928 20:22:48.678570 19088 leader_election.cc:288] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 20:22:48.678615 19092 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb [term 2 FOLLOWER]: Leader election won for term 2
I20260928 20:22:48.678728 19092 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb [term 2 LEADER]: Becoming Leader. State: Replica: fe5679c7c74f4b3e843599dc347a86bb, State: Running, Role: LEADER
I20260928 20:22:48.678848 19088 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:22:48.678869 19092 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe5679c7c74f4b3e843599dc347a86bb" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 40865 } }
I20260928 20:22:48.679224 19093 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "fe5679c7c74f4b3e843599dc347a86bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe5679c7c74f4b3e843599dc347a86bb" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 40865 } } }
I20260928 20:22:48.679311 19093 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb [sys.catalog]: This master's current role is: LEADER
I20260928 20:22:48.679594 19094 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe5679c7c74f4b3e843599dc347a86bb. Latest consensus state: current_term: 2 leader_uuid: "fe5679c7c74f4b3e843599dc347a86bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe5679c7c74f4b3e843599dc347a86bb" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 40865 } } }
I20260928 20:22:48.679685 19094 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb [sys.catalog]: This master's current role is: LEADER
I20260928 20:22:48.679977 19098 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:22:48.680794 19098 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:22:48.680938 19098 catalog_manager.cc:1259] Loaded cluster ID: a6286dddff7642f1866ae832f5093957
I20260928 20:22:48.681171 19098 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:22:48.682363 19098 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:22:48.682542 19098 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb: Loaded TSK: 0
I20260928 20:22:48.691767 19098 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb: Generated new TSK 1
I20260928 20:22:48.691895 19098 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 20:22:48.754206 19090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:22:48.754464 19090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:22:48.754509 19090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:22:48.755889 19090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:22:48.755957 19090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:22:48.756002 19090 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.1
I20260928 20:22:48.757601 19090 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:36695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab
--principal=kudu/127.3.18.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.1:34605
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.18.1
--webserver_port=41063
--tserver_master_addrs=127.3.18.62:40865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.1
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:22:48.757889 19090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:22:48.758142 19090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:22:48.764535 19117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:22:48.764535 19116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:22:48.764925 19090 server_base.cc:1034] running on GCE node
W20260928 20:22:48.764639 19119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:22:48.765223 19090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:22:48.765527 19090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:22:48.766718 19090 hybrid_clock.cc:648] HybridClock initialized: now 1790626968766716 us; error 46 us; skew 500 ppm
Sep 28 20:22:48 dist-test-slave-b83v krb5kdc[18472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626968, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:22:48.769623 19090 init.cc:377] Logged in from keytab as kudu/127.3.18.1@KRBTEST.COM (short username kudu)
I20260928 20:22:48.770222 19090 webserver.cc:460] Webserver started at http://127.3.18.1:41063/ using document root <none> and password file <none>
I20260928 20:22:48.770479 19090 fs_manager.cc:359] Metadata directory not provided
I20260928 20:22:48.770545 19090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:22:48.772199 19090 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:22:48.773105 19126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:22:48.773288 19090 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:22:48.773415 19090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-0/wal
uuid: "1073bb7be04e4025bd11239f3303b0e3"
format_stamp: "Formatted at 2026-09-28 20:22:28 on dist-test-slave-b83v"
server_key: "5f970228296b9f459911d0b49a1c54d3"
server_key_iv: "290dac0c00c973deaf9703386fdfbd18"
server_key_version: "encryptionkey@0"
I20260928 20:22:48.773842 19090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:22:48.784662 19090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:22:48.787122 19090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:22:48.787289 19090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:22:48.787588 19090 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:22:48.788148 19133 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 20:22:48.789845 19090 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 20:22:48.789904 19090 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 20:22:48.789932 19090 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 20:22:48.790675 19090 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 20:22:48.790721 19090 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 20:22:48.790771 19133 tablet_bootstrap.cc:492] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 1073bb7be04e4025bd11239f3303b0e3: Bootstrap starting.
I20260928 20:22:48.798923 19090 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.1:34605
I20260928 20:22:48.799113 19240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.1:34605 every 8 connection(s)
I20260928 20:22:48.799512 19090 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
I20260928 20:22:48.802446 19133 log.cc:824] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 1073bb7be04e4025bd11239f3303b0e3: Log is configured to *not* fsync() on all Append() calls
I20260928 20:22:48.804935 19133 tablet_bootstrap.cc:492] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 1073bb7be04e4025bd11239f3303b0e3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 20:22:48.805274 19133 tablet_bootstrap.cc:492] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 1073bb7be04e4025bd11239f3303b0e3: Bootstrap complete.
I20260928 20:22:48.806340 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 19090
I20260928 20:22:48.806468 19133 ts_tablet_manager.cc:1397] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 1073bb7be04e4025bd11239f3303b0e3: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.009s
I20260928 20:22:48.806655 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.2:43745
--local_ip_for_outbound_sockets=127.3.18.2
--tserver_master_addrs=127.3.18.62:40865
--webserver_port=44901
--webserver_interface=127.3.18.2
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab
--principal=kudu/127.3.18.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:36695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 20:22:48.807883 19133 raft_consensus.cc:348] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 1073bb7be04e4025bd11239f3303b0e3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891977fbea749aeb0f0e9e80b4f9361" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 43745 } } peers { permanent_uuid: "3638055a7dc94f378d5f16baf99f151d" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 33365 } } peers { permanent_uuid: "1073bb7be04e4025bd11239f3303b0e3" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34605 } }
I20260928 20:22:48.808243 19133 raft_consensus.cc:729] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 1073bb7be04e4025bd11239f3303b0e3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1073bb7be04e4025bd11239f3303b0e3, State: Initialized, Role: FOLLOWER
I20260928 20:22:48.808426 19133 consensus_queue.cc:260] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 1073bb7be04e4025bd11239f3303b0e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891977fbea749aeb0f0e9e80b4f9361" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 43745 } } peers { permanent_uuid: "3638055a7dc94f378d5f16baf99f151d" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 33365 } } peers { permanent_uuid: "1073bb7be04e4025bd11239f3303b0e3" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34605 } }
I20260928 20:22:48.808729 19133 ts_tablet_manager.cc:1428] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 1073bb7be04e4025bd11239f3303b0e3: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
Sep 28 20:22:48 dist-test-slave-b83v krb5kdc[18472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626968, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.1@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:22:48.816461 19243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:48.800366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40865 (local address 127.3.18.1:57377)
0928 20:22:48.802670 (+ 2304us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:48.802736 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0928 20:22:48.803387 (+ 651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:22:48.803974 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:22:48.803978 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:22:48.804336 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:22:48.804608 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:48.804615 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:48.806440 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:48.806441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:48.807699 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:48.807702 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:48.808754 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:48.808755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:48.808891 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:48.809581 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:22:48.809592 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:22:48.811520 (+ 1928us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:22:48.814375 (+ 2855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:22:48.814385 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:22:48.814387 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:22:48.814693 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:22:48.815017 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:22:48.815018 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:22:48.815018 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:22:48.815152 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:22:48.815693 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:22:48.815698 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:22:48.815904 (+ 206us) client_negotiation.cc:769] Sending connection context
0928 20:22:48.816015 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 20:22:48.816235 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2251,"thread_start_us":96,"threads_started":1}
I20260928 20:22:48.817334 19244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:48.800509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:57377 (local address 127.3.18.62:40865)
0928 20:22:48.800730 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:48.800740 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:48.802907 (+ 2167us) server_negotiation.cc:407] Connection header received
0928 20:22:48.803652 (+ 745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:48.803660 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:48.803733 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:48.803923 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:22:48.804754 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:48.806303 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:48.807841 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:48.808527 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:48.808628 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:48.811648 (+ 3020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:22:48.811653 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:22:48.811653 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:22:48.811669 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:22:48.814272 (+ 2603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:48.814790 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:48.814795 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:48.814795 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:48.814933 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:48.815220 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:48.815221 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:48.815222 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:48.815591 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:22:48.815707 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:48.816107 (+ 400us) server_negotiation.cc:299] Negotiation successful
0928 20:22:48.816282 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":104,"threads_started":1}
I20260928 20:22:48.818005 19241 heartbeater.cc:347] Connected to a master server at 127.3.18.62:40865
I20260928 20:22:48.818109 19241 heartbeater.cc:464] Registering TS with master...
I20260928 20:22:48.818290 19241 heartbeater.cc:511] Master 127.3.18.62:40865 requested a full tablet report, sending...
I20260928 20:22:48.819132 19052 ts_manager.cc:194] Registered new tserver with Master: 1073bb7be04e4025bd11239f3303b0e3 (127.3.18.1:34605)
I20260928 20:22:48.820482 19052 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.18.1@KRBTEST.COM'} at 127.3.18.1:57377
I20260928 20:22:48.821381 19241 heartbeater.cc:503] Master 127.3.18.62:40865 was elected leader, sending a full tablet report...
Sep 28 20:22:48 dist-test-slave-b83v krb5kdc[18472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626968, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.62@KRBTEST.COM for kudu/127.3.18.1@KRBTEST.COM
I20260928 20:22:48.828855 19248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:48.819918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.1:34605 (local address 127.0.0.1:59476)
0928 20:22:48.820213 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:48.820238 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 20:22:48.820327 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:22:48.820505 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:22:48.820509 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:22:48.820866 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:22:48.820945 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:48.820948 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:48.822400 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:48.822401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:48.822980 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:48.822983 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:48.823601 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:48.823604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:48.823703 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:48.824269 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:22:48.824281 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:22:48.825513 (+ 1232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:22:48.827670 (+ 2157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:22:48.827678 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:22:48.827680 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:22:48.827928 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:22:48.828132 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:22:48.828133 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:22:48.828133 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:22:48.828218 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:22:48.828555 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:22:48.828561 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:22:48.828609 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 20:22:48.828687 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 20:22:48.828783 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":111,"threads_started":1}
I20260928 20:22:48.828931 19249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:48.819977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59476 (local address 127.3.18.1:34605)
0928 20:22:48.820159 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:48.820162 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:48.820307 (+ 145us) server_negotiation.cc:407] Connection header received
0928 20:22:48.820425 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:48.820427 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:48.820443 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:48.820503 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:22:48.821023 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:48.822325 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:48.823056 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:48.823518 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:48.823592 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:48.825587 (+ 1995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:22:48.825595 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:22:48.825596 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:22:48.825603 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:22:48.827525 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:48.828000 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:48.828003 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:48.828004 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:48.828069 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:48.828276 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:48.828277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:48.828277 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:48.828484 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:22:48.828576 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:48.828736 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 20:22:48.828833 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":147,"thread_start_us":79,"threads_started":1}
I20260928 20:22:48.829317 19175 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d8762e30b5c4b8398cb3f0fcc075ab5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:22:28 UTC) from {username='kudu', principal='kudu/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:59476
I20260928 20:22:48.829890 19250 tablet_replica.cc:330] stopping tablet replica
I20260928 20:22:48.829998 19250 raft_consensus.cc:2229] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 1073bb7be04e4025bd11239f3303b0e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:22:48.830068 19250 raft_consensus.cc:2258] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 1073bb7be04e4025bd11239f3303b0e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:22:48.830364 19250 ts_tablet_manager.cc:1905] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 1073bb7be04e4025bd11239f3303b0e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:22:48.832557 19250 ts_tablet_manager.cc:1918] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 1073bb7be04e4025bd11239f3303b0e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:22:48.832700 19250 log.cc:1196] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 1073bb7be04e4025bd11239f3303b0e3: Deleting WAL directory at /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-0/wal/wals/6d8762e30b5c4b8398cb3f0fcc075ab5
I20260928 20:22:48.832926 19250 ts_tablet_manager.cc:1939] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 1073bb7be04e4025bd11239f3303b0e3: Deleting consensus metadata
I20260928 20:22:48.833469 19037 catalog_manager.cc:4841] TS 1073bb7be04e4025bd11239f3303b0e3 (127.3.18.1:34605): tablet 6d8762e30b5c4b8398cb3f0fcc075ab5 (table test-table [id=93e1921d6ef94edb9b7e3ab1fcfd7754]) successfully deleted
W20260928 20:22:48.897423 19246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:22:48.897692 19246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:22:48.897737 19246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:22:48.898938 19246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:22:48.899019 19246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:22:48.899075 19246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.2
I20260928 20:22:48.900643 19246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:36695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab
--principal=kudu/127.3.18.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.2:43745
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.18.2
--webserver_port=44901
--tserver_master_addrs=127.3.18.62:40865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19246
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.2
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:22:48.900930 19246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:22:48.901182 19246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:22:48.906774 19258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:22:48.907013 19256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:22:48.906973 19246 server_base.cc:1034] running on GCE node
W20260928 20:22:48.906773 19255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:22:48.907397 19246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:22:48.907732 19246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:22:48.908969 19246 hybrid_clock.cc:648] HybridClock initialized: now 1790626968908959 us; error 32 us; skew 500 ppm
Sep 28 20:22:48 dist-test-slave-b83v krb5kdc[18472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626968, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:22:48.911867 19246 init.cc:377] Logged in from keytab as kudu/127.3.18.2@KRBTEST.COM (short username kudu)
I20260928 20:22:48.912395 19246 webserver.cc:460] Webserver started at http://127.3.18.2:44901/ using document root <none> and password file <none>
I20260928 20:22:48.912665 19246 fs_manager.cc:359] Metadata directory not provided
I20260928 20:22:48.912747 19246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:22:48.914212 19246 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 20:22:48.915000 19265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:22:48.915241 19246 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:22:48.915350 19246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-1/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-1/wal
uuid: "9891977fbea749aeb0f0e9e80b4f9361"
format_stamp: "Formatted at 2026-09-28 20:22:28 on dist-test-slave-b83v"
server_key: "0c19841604e3953e1aedad39d14244a9"
server_key_iv: "3300c18e4170e2c69334ff0566b0733a"
server_key_version: "encryptionkey@0"
I20260928 20:22:48.915714 19246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:22:48.935515 19246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:22:48.938264 19246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:22:48.938480 19246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:22:48.938803 19246 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:22:48.939322 19272 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 20:22:48.940976 19246 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 20:22:48.941088 19246 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 20:22:48.941130 19246 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 20:22:48.941881 19246 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 20:22:48.941944 19246 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 20:22:48.942055 19272 tablet_bootstrap.cc:492] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 9891977fbea749aeb0f0e9e80b4f9361: Bootstrap starting.
I20260928 20:22:48.949326 19246 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.2:43745
I20260928 20:22:48.949394 19379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.2:43745 every 8 connection(s)
I20260928 20:22:48.949842 19246 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
I20260928 20:22:48.951999 19272 log.cc:824] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 9891977fbea749aeb0f0e9e80b4f9361: Log is configured to *not* fsync() on all Append() calls
I20260928 20:22:48.953806 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 19246
I20260928 20:22:48.954327 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.3:33365
--local_ip_for_outbound_sockets=127.3.18.3
--tserver_master_addrs=127.3.18.62:40865
--webserver_port=38473
--webserver_interface=127.3.18.3
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab
--principal=kudu/127.3.18.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:36695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 20:22:48.954885 19272 tablet_bootstrap.cc:492] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 9891977fbea749aeb0f0e9e80b4f9361: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 20:22:48.955217 19272 tablet_bootstrap.cc:492] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 9891977fbea749aeb0f0e9e80b4f9361: Bootstrap complete.
I20260928 20:22:48.956548 19272 ts_tablet_manager.cc:1397] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 9891977fbea749aeb0f0e9e80b4f9361: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.009s
Sep 28 20:22:48 dist-test-slave-b83v krb5kdc[18472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626968, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.2@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:22:48.958830 19272 raft_consensus.cc:348] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 9891977fbea749aeb0f0e9e80b4f9361 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891977fbea749aeb0f0e9e80b4f9361" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 43745 } } peers { permanent_uuid: "3638055a7dc94f378d5f16baf99f151d" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 33365 } } peers { permanent_uuid: "1073bb7be04e4025bd11239f3303b0e3" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34605 } }
I20260928 20:22:48.959136 19272 raft_consensus.cc:729] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 9891977fbea749aeb0f0e9e80b4f9361 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9891977fbea749aeb0f0e9e80b4f9361, State: Initialized, Role: FOLLOWER
I20260928 20:22:48.959282 19272 consensus_queue.cc:260] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 9891977fbea749aeb0f0e9e80b4f9361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9891977fbea749aeb0f0e9e80b4f9361" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 43745 } } peers { permanent_uuid: "3638055a7dc94f378d5f16baf99f151d" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 33365 } } peers { permanent_uuid: "1073bb7be04e4025bd11239f3303b0e3" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34605 } }
I20260928 20:22:48.963100 19244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:48.950696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:51863 (local address 127.3.18.62:40865)
0928 20:22:48.950759 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:48.950761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:48.951090 (+ 329us) server_negotiation.cc:407] Connection header received
0928 20:22:48.951883 (+ 793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:48.951885 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:48.951900 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:48.951958 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:22:48.952802 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:48.953756 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:48.955668 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:48.956045 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:48.956093 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:48.959086 (+ 2993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:22:48.959088 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:22:48.959089 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:22:48.959097 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:22:48.961133 (+ 2036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:48.961687 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:48.961689 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:48.961690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:48.961753 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:48.962157 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:48.962158 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:48.962159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:48.962350 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:22:48.962399 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:48.962838 (+ 439us) server_negotiation.cc:299] Negotiation successful
0928 20:22:48.962958 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 20:22:48.963755 19382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:48.950616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40865 (local address 127.3.18.2:51863)
0928 20:22:48.950918 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:48.950968 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 20:22:48.951741 (+ 773us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:22:48.952035 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:22:48.952039 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:22:48.952368 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:22:48.952666 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:48.952673 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:48.953945 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:48.953947 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:48.955548 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:48.955553 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:48.956191 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:48.956192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:48.956378 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:48.957087 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:22:48.957117 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:22:48.958965 (+ 1848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:22:48.961251 (+ 2286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:22:48.961258 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:22:48.961260 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:22:48.961586 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:22:48.961831 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:22:48.961832 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:22:48.961832 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:22:48.961978 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:22:48.962462 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:22:48.962469 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:22:48.962657 (+ 188us) client_negotiation.cc:769] Sending connection context
0928 20:22:48.963234 (+ 577us) client_negotiation.cc:240] Negotiation successful
0928 20:22:48.963461 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":92,"threads_started":1}
I20260928 20:22:48.964061 19272 ts_tablet_manager.cc:1428] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 9891977fbea749aeb0f0e9e80b4f9361: Time spent starting tablet: real 0.007s user 0.001s sys 0.002s
I20260928 20:22:48.964340 19380 heartbeater.cc:347] Connected to a master server at 127.3.18.62:40865
I20260928 20:22:48.964450 19380 heartbeater.cc:464] Registering TS with master...
I20260928 20:22:48.964627 19380 heartbeater.cc:511] Master 127.3.18.62:40865 requested a full tablet report, sending...
I20260928 20:22:48.965229 19052 ts_manager.cc:194] Registered new tserver with Master: 9891977fbea749aeb0f0e9e80b4f9361 (127.3.18.2:43745)
I20260928 20:22:48.965929 19052 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.18.2@KRBTEST.COM'} at 127.3.18.2:51863
I20260928 20:22:48.967327 19380 heartbeater.cc:503] Master 127.3.18.62:40865 was elected leader, sending a full tablet report...
Sep 28 20:22:48 dist-test-slave-b83v krb5kdc[18472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626968, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.62@KRBTEST.COM for kudu/127.3.18.2@KRBTEST.COM
I20260928 20:22:48.974085 19248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:48.965631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.2:43745 (local address 127.0.0.1:41628)
0928 20:22:48.965703 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:48.965720 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 20:22:48.965761 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:22:48.966406 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:22:48.966407 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:22:48.966624 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:22:48.966669 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:48.966672 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:48.967904 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:48.967905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:48.968535 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:48.968538 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:48.969120 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:48.969124 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:48.969205 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:48.969717 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:22:48.969723 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:22:48.970922 (+ 1199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:22:48.972957 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:22:48.972959 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:22:48.972959 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:22:48.973142 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:22:48.973374 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:22:48.973375 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:22:48.973375 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:22:48.973437 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:22:48.973739 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:22:48.973740 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:22:48.973790 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 20:22:48.973857 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 20:22:48.973981 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 20:22:48.974092 19386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:48.966095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41628 (local address 127.3.18.2:43745)
0928 20:22:48.966254 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:48.966258 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:48.966268 (+ 10us) server_negotiation.cc:407] Connection header received
0928 20:22:48.966298 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:48.966300 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:48.966313 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:48.966401 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:22:48.966764 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:48.967815 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:48.968691 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:48.968994 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:48.969089 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:48.971033 (+ 1944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:22:48.971042 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:22:48.971042 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:22:48.971050 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:22:48.972844 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:48.973238 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:48.973242 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:48.973242 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:48.973303 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:48.973508 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:48.973509 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:48.973510 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:48.973668 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:22:48.973760 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:48.973899 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 20:22:48.973992 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":96,"threads_started":1}
I20260928 20:22:48.974687 19314 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d8762e30b5c4b8398cb3f0fcc075ab5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:22:28 UTC) from {username='kudu', principal='kudu/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:41628
I20260928 20:22:48.975222 19387 tablet_replica.cc:330] stopping tablet replica
I20260928 20:22:48.975338 19387 raft_consensus.cc:2229] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 9891977fbea749aeb0f0e9e80b4f9361 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:22:48.975427 19387 raft_consensus.cc:2258] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 9891977fbea749aeb0f0e9e80b4f9361 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:22:48.975797 19387 ts_tablet_manager.cc:1905] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 9891977fbea749aeb0f0e9e80b4f9361: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:22:48.978910 19387 ts_tablet_manager.cc:1918] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 9891977fbea749aeb0f0e9e80b4f9361: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:22:48.978996 19387 log.cc:1196] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 9891977fbea749aeb0f0e9e80b4f9361: Deleting WAL directory at /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-1/wal/wals/6d8762e30b5c4b8398cb3f0fcc075ab5
I20260928 20:22:48.979269 19387 ts_tablet_manager.cc:1939] T 6d8762e30b5c4b8398cb3f0fcc075ab5 P 9891977fbea749aeb0f0e9e80b4f9361: Deleting consensus metadata
I20260928 20:22:48.979957 19036 catalog_manager.cc:4841] TS 9891977fbea749aeb0f0e9e80b4f9361 (127.3.18.2:43745): tablet 6d8762e30b5c4b8398cb3f0fcc075ab5 (table test-table [id=93e1921d6ef94edb9b7e3ab1fcfd7754]) successfully deleted
W20260928 20:22:49.045953 19384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:22:49.046190 19384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:22:49.046216 19384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:22:49.047425 19384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:22:49.047482 19384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:22:49.047509 19384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.3
I20260928 20:22:49.049163 19384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:36695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab
--principal=kudu/127.3.18.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.3:33365
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.18.3
--webserver_port=38473
--tserver_master_addrs=127.3.18.62:40865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19384
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.3
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:22:49.049456 19384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:22:49.049759 19384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:22:49.055239 19392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:22:49.055284 19384 server_base.cc:1034] running on GCE node
W20260928 20:22:49.055225 19395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:22:49.055225 19393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:22:49.055835 19384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:22:49.056173 19384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:22:49.057360 19384 hybrid_clock.cc:648] HybridClock initialized: now 1790626969057345 us; error 33 us; skew 500 ppm
Sep 28 20:22:49 dist-test-slave-b83v krb5kdc[18472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626969, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:22:49.060325 19384 init.cc:377] Logged in from keytab as kudu/127.3.18.3@KRBTEST.COM (short username kudu)
I20260928 20:22:49.060796 19384 webserver.cc:460] Webserver started at http://127.3.18.3:38473/ using document root <none> and password file <none>
I20260928 20:22:49.060995 19384 fs_manager.cc:359] Metadata directory not provided
I20260928 20:22:49.061053 19384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:22:49.062410 19384 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:22:49.063112 19402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:22:49.063333 19384 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:22:49.063450 19384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-2/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-2/wal
uuid: "3638055a7dc94f378d5f16baf99f151d"
format_stamp: "Formatted at 2026-09-28 20:22:28 on dist-test-slave-b83v"
server_key: "03c355bf2c34de2ec07ddabb3293d980"
server_key_iv: "c64512be4d15bbebfff90112b37c8c6b"
server_key_version: "encryptionkey@0"
I20260928 20:22:49.063942 19384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:22:49.073141 19384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:22:49.075544 19384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:22:49.075718 19384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:22:49.075999 19384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:22:49.076380 19384 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:22:49.076448 19384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:22:49.076491 19384 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:22:49.076519 19384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:22:49.082580 19384 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.3:33365
I20260928 20:22:49.082666 19515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.3:33365 every 8 connection(s)
I20260928 20:22:49.083031 19384 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
I20260928 20:22:49.083333 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 19384
Sep 28 20:22:49 dist-test-slave-b83v krb5kdc[18472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626969, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.3@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:22:49.090986 19244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:49.083869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43418 (local address 127.3.18.62:40865)
0928 20:22:49.083930 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:49.083933 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:49.084164 (+ 231us) server_negotiation.cc:407] Connection header received
0928 20:22:49.084245 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:49.084246 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:49.084257 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:49.084311 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:22:49.084948 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:49.085433 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:49.086208 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:49.086418 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:49.086494 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:49.087799 (+ 1305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:22:49.087801 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:22:49.087802 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:22:49.087809 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:22:49.089563 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:49.089969 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:49.089970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:49.089971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:49.090021 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:49.090238 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:49.090239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:49.090239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:49.090329 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:22:49.090443 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:49.090772 (+ 329us) server_negotiation.cc:299] Negotiation successful
0928 20:22:49.090863 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 20:22:49.093458 19519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:49.083897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:35267 (local address 127.3.18.62:40865)
0928 20:22:49.084048 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:49.084050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:49.084334 (+ 284us) server_negotiation.cc:407] Connection header received
0928 20:22:49.084852 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:49.084853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:49.084865 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:49.084929 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:22:49.085903 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:49.086418 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:49.087439 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:49.087660 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:49.087803 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:49.090371 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:22:49.090372 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:22:49.090372 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:22:49.090379 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:22:49.092132 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:49.092515 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:49.092516 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:49.092516 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:49.092578 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:49.092817 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:49.092818 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:49.092818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:49.092924 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:22:49.092981 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:49.093313 (+ 332us) server_negotiation.cc:299] Negotiation successful
0928 20:22:49.093389 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":134,"thread_start_us":64,"threads_started":1}
I20260928 20:22:49.093636 19520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:49.083885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40865 (local address 127.3.18.3:35267)
0928 20:22:49.084196 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:49.084251 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 20:22:49.084749 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:22:49.085004 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:22:49.085008 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:22:49.085614 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:22:49.085802 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:49.085807 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:49.086535 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:49.086536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:49.087344 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:49.087347 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:49.087817 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:49.087819 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:49.087947 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:49.088581 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:22:49.088590 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:22:49.090255 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:22:49.092205 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:22:49.092211 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:22:49.092213 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:22:49.092429 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:22:49.092655 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:22:49.092656 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:22:49.092656 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:22:49.092753 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:22:49.093018 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:22:49.093025 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:22:49.093156 (+ 131us) client_negotiation.cc:769] Sending connection context
0928 20:22:49.093267 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 20:22:49.093398 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":89,"threads_started":1}
I20260928 20:22:49.094187 19516 heartbeater.cc:347] Connected to a master server at 127.3.18.62:40865
I20260928 20:22:49.094309 19516 heartbeater.cc:464] Registering TS with master...
I20260928 20:22:49.094493 19516 heartbeater.cc:511] Master 127.3.18.62:40865 requested a full tablet report, sending...
I20260928 20:22:49.094906 19052 ts_manager.cc:194] Registered new tserver with Master: 3638055a7dc94f378d5f16baf99f151d (127.3.18.3:33365)
I20260928 20:22:49.095280 19052 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.18.3@KRBTEST.COM'} at 127.3.18.3:35267
I20260928 20:22:49.102546 3144 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 20:22:49.105465 19519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:49.102958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43420 (local address 127.3.18.62:40865)
0928 20:22:49.103021 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:49.103027 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:49.103121 (+ 94us) server_negotiation.cc:407] Connection header received
0928 20:22:49.103187 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:49.103188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:49.103223 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:49.103291 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:22:49.103377 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:49.103940 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:49.104812 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:49.105000 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:49.105050 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:49.105241 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:22:49.105284 (+ 43us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 20:22:49.105417 (+ 133us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43420: token expired
Metrics: {"server-negotiator.queue_time_us":36}
W20260928 20:22:49.105537 19519 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43420: token expired
W20260928 20:22:49.105650 18980 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.3.18.62:40865)
I20260928 20:22:49.111910 19519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:49.105913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43436 (local address 127.3.18.62:40865)
0928 20:22:49.105974 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:49.105975 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:49.105983 (+ 8us) server_negotiation.cc:407] Connection header received
0928 20:22:49.106015 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:49.106016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:49.106052 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:49.106090 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:22:49.106637 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:49.107068 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:49.107871 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:49.108062 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:49.108125 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:49.109344 (+ 1219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:22:49.109346 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:22:49.109346 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:22:49.109352 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:22:49.110618 (+ 1266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:49.111014 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:49.111015 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:49.111015 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:49.111077 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:22:49.111313 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:22:49.111314 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:22:49.111314 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:22:49.111412 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:22:49.111490 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:49.111684 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 20:22:49.111753 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 20:22:49.113304 19052 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43436:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 20:22:49.115096 19052 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 20:22:49.117381 19175 tablet_service.cc:1467] Processing CreateTablet for tablet ae59f4c4b8344f5582f207387f028714 (DEFAULT_TABLE table=test-table [id=b4bc838b59e2441998b4667af4cb3e1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:22:49.117468 19314 tablet_service.cc:1467] Processing CreateTablet for tablet ae59f4c4b8344f5582f207387f028714 (DEFAULT_TABLE table=test-table [id=b4bc838b59e2441998b4667af4cb3e1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:22:49.117604 19314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae59f4c4b8344f5582f207387f028714. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:22:49.117595 19175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae59f4c4b8344f5582f207387f028714. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:22:49.119515 19133 tablet_bootstrap.cc:492] T ae59f4c4b8344f5582f207387f028714 P 1073bb7be04e4025bd11239f3303b0e3: Bootstrap starting.
I20260928 20:22:49.119918 19133 tablet_bootstrap.cc:654] T ae59f4c4b8344f5582f207387f028714 P 1073bb7be04e4025bd11239f3303b0e3: Neither blocks nor log segments found. Creating new log.
I20260928 20:22:49.120473 19272 tablet_bootstrap.cc:492] T ae59f4c4b8344f5582f207387f028714 P 9891977fbea749aeb0f0e9e80b4f9361: Bootstrap starting.
I20260928 20:22:49.120512 19133 tablet_bootstrap.cc:492] T ae59f4c4b8344f5582f207387f028714 P 1073bb7be04e4025bd11239f3303b0e3: No bootstrap required, opened a new log
I20260928 20:22:49.120560 19133 ts_tablet_manager.cc:1397] T ae59f4c4b8344f5582f207387f028714 P 1073bb7be04e4025bd11239f3303b0e3: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 20:22:49.120783 19272 tablet_bootstrap.cc:654] T ae59f4c4b8344f5582f207387f028714 P 9891977fbea749aeb0f0e9e80b4f9361: Neither blocks nor log segments found. Creating new log.
I20260928 20:22:49.120841 19133 raft_consensus.cc:348] T ae59f4c4b8344f5582f207387f028714 P 1073bb7be04e4025bd11239f3303b0e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3638055a7dc94f378d5f16baf99f151d" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 33365 } } peers { permanent_uuid: "1073bb7be04e4025bd11239f3303b0e3" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34605 } } peers { permanent_uuid: "9891977fbea749aeb0f0e9e80b4f9361" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 43745 } }
I20260928 20:22:49.120891 19133 raft_consensus.cc:374] T ae59f4c4b8344f5582f207387f028714 P 1073bb7be04e4025bd11239f3303b0e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:22:49.120918 19133 raft_consensus.cc:729] T ae59f4c4b8344f5582f207387f028714 P 1073bb7be04e4025bd11239f3303b0e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1073bb7be04e4025bd11239f3303b0e3, State: Initialized, Role: FOLLOWER
I20260928 20:22:49.120980 19133 consensus_queue.cc:260] T ae59f4c4b8344f5582f207387f028714 P 1073bb7be04e4025bd11239f3303b0e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3638055a7dc94f378d5f16baf99f151d" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 33365 } } peers { permanent_uuid: "1073bb7be04e4025bd11239f3303b0e3" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34605 } } peers { permanent_uuid: "9891977fbea749aeb0f0e9e80b4f9361" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 43745 } }
I20260928 20:22:49.121074 19133 ts_tablet_manager.cc:1428] T ae59f4c4b8344f5582f207387f028714 P 1073bb7be04e4025bd11239f3303b0e3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 20:22:49.121639 19272 tablet_bootstrap.cc:492] T ae59f4c4b8344f5582f207387f028714 P 9891977fbea749aeb0f0e9e80b4f9361: No bootstrap required, opened a new log
I20260928 20:22:49.121696 19272 ts_tablet_manager.cc:1397] T ae59f4c4b8344f5582f207387f028714 P 9891977fbea749aeb0f0e9e80b4f9361: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260928 20:22:49.121836 19272 raft_consensus.cc:348] T ae59f4c4b8344f5582f207387f028714 P 9891977fbea749aeb0f0e9e80b4f9361 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3638055a7dc94f378d5f16baf99f151d" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 33365 } } peers { permanent_uuid: "1073bb7be04e4025bd11239f3303b0e3" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34605 } } peers { permanent_uuid: "9891977fbea749aeb0f0e9e80b4f9361" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 43745 } }
I20260928 20:22:49.121907 19272 raft_consensus.cc:374] T ae59f4c4b8344f5582f207387f028714 P 9891977fbea749aeb0f0e9e80b4f9361 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:22:49.121956 19272 raft_consensus.cc:729] T ae59f4c4b8344f5582f207387f028714 P 9891977fbea749aeb0f0e9e80b4f9361 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9891977fbea749aeb0f0e9e80b4f9361, State: Initialized, Role: FOLLOWER
I20260928 20:22:49.122006 19272 consensus_queue.cc:260] T ae59f4c4b8344f5582f207387f028714 P 9891977fbea749aeb0f0e9e80b4f9361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3638055a7dc94f378d5f16baf99f151d" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 33365 } } peers { permanent_uuid: "1073bb7be04e4025bd11239f3303b0e3" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34605 } } peers { permanent_uuid: "9891977fbea749aeb0f0e9e80b4f9361" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 43745 } }
I20260928 20:22:49.122095 19272 ts_tablet_manager.cc:1428] T ae59f4c4b8344f5582f207387f028714 P 9891977fbea749aeb0f0e9e80b4f9361: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 20:22:49.123956 19248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:49.117171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.3:33365 (local address 127.0.0.1:36864)
0928 20:22:49.117255 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:49.117267 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:22:49.117304 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:22:49.119779 (+ 2475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:22:49.119780 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:22:49.119782 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:22:49.119842 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:49.119844 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:49.121582 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:49.121584 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:49.122814 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:49.122818 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:49.123621 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:49.123623 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:49.123688 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:49.123708 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 20:22:49.123833 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 20:22:49.123889 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260928 20:22:49.123966 19522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:49.119279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36864 (local address 127.3.18.3:33365)
0928 20:22:49.119501 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:49.119509 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:49.119522 (+ 13us) server_negotiation.cc:407] Connection header received
0928 20:22:49.119603 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:49.119605 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:49.119626 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:49.119927 (+ 301us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:22:49.120044 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:49.121505 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:49.122897 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:49.123520 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:49.123592 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:49.123646 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:49.123850 (+ 204us) server_negotiation.cc:299] Negotiation successful
0928 20:22:49.123897 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":124,"threads_started":1}
I20260928 20:22:49.124850 19450 tablet_service.cc:1467] Processing CreateTablet for tablet ae59f4c4b8344f5582f207387f028714 (DEFAULT_TABLE table=test-table [id=b4bc838b59e2441998b4667af4cb3e1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:22:49.125288 19450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae59f4c4b8344f5582f207387f028714. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:22:49.128664 19525 tablet_bootstrap.cc:492] T ae59f4c4b8344f5582f207387f028714 P 3638055a7dc94f378d5f16baf99f151d: Bootstrap starting.
I20260928 20:22:49.129360 19525 tablet_bootstrap.cc:654] T ae59f4c4b8344f5582f207387f028714 P 3638055a7dc94f378d5f16baf99f151d: Neither blocks nor log segments found. Creating new log.
I20260928 20:22:49.129895 19525 log.cc:824] T ae59f4c4b8344f5582f207387f028714 P 3638055a7dc94f378d5f16baf99f151d: Log is configured to *not* fsync() on all Append() calls
I20260928 20:22:49.130731 19525 tablet_bootstrap.cc:492] T ae59f4c4b8344f5582f207387f028714 P 3638055a7dc94f378d5f16baf99f151d: No bootstrap required, opened a new log
I20260928 20:22:49.130800 19525 ts_tablet_manager.cc:1397] T ae59f4c4b8344f5582f207387f028714 P 3638055a7dc94f378d5f16baf99f151d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 20:22:49.132589 19525 raft_consensus.cc:348] T ae59f4c4b8344f5582f207387f028714 P 3638055a7dc94f378d5f16baf99f151d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3638055a7dc94f378d5f16baf99f151d" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 33365 } } peers { permanent_uuid: "1073bb7be04e4025bd11239f3303b0e3" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34605 } } peers { permanent_uuid: "9891977fbea749aeb0f0e9e80b4f9361" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 43745 } }
I20260928 20:22:49.132762 19525 raft_consensus.cc:374] T ae59f4c4b8344f5582f207387f028714 P 3638055a7dc94f378d5f16baf99f151d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:22:49.132786 19525 raft_consensus.cc:729] T ae59f4c4b8344f5582f207387f028714 P 3638055a7dc94f378d5f16baf99f151d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3638055a7dc94f378d5f16baf99f151d, State: Initialized, Role: FOLLOWER
I20260928 20:22:49.132925 19525 consensus_queue.cc:260] T ae59f4c4b8344f5582f207387f028714 P 3638055a7dc94f378d5f16baf99f151d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3638055a7dc94f378d5f16baf99f151d" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 33365 } } peers { permanent_uuid: "1073bb7be04e4025bd11239f3303b0e3" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34605 } } peers { permanent_uuid: "9891977fbea749aeb0f0e9e80b4f9361" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 43745 } }
I20260928 20:22:49.133260 19525 ts_tablet_manager.cc:1428] T ae59f4c4b8344f5582f207387f028714 P 3638055a7dc94f378d5f16baf99f151d: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 20:22:49.133327 19516 heartbeater.cc:503] Master 127.3.18.62:40865 was elected leader, sending a full tablet report...
W20260928 20:22:49.200511 19381 tablet.cc:2369] T ae59f4c4b8344f5582f207387f028714 P 9891977fbea749aeb0f0e9e80b4f9361: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 20:22:49.242766 19385 raft_consensus.cc:482] T ae59f4c4b8344f5582f207387f028714 P 9891977fbea749aeb0f0e9e80b4f9361 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:22:49.242913 19385 raft_consensus.cc:504] T ae59f4c4b8344f5582f207387f028714 P 9891977fbea749aeb0f0e9e80b4f9361 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3638055a7dc94f378d5f16baf99f151d" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 33365 } } peers { permanent_uuid: "1073bb7be04e4025bd11239f3303b0e3" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34605 } } peers { permanent_uuid: "9891977fbea749aeb0f0e9e80b4f9361" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 43745 } }
I20260928 20:22:49.243319 19385 leader_election.cc:288] T ae59f4c4b8344f5582f207387f028714 P 9891977fbea749aeb0f0e9e80b4f9361 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3638055a7dc94f378d5f16baf99f151d (127.3.18.3:33365), 1073bb7be04e4025bd11239f3303b0e3 (127.3.18.1:34605)
I20260928 20:22:49.247620 19522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:49.243705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:34365 (local address 127.3.18.3:33365)
0928 20:22:49.243749 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:49.243752 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:49.243785 (+ 33us) server_negotiation.cc:407] Connection header received
0928 20:22:49.243818 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:49.243819 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:49.243835 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:49.243914 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:22:49.244167 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:49.244909 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:49.246506 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:49.247153 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:49.247235 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:49.247248 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:49.247484 (+ 236us) server_negotiation.cc:299] Negotiation successful
0928 20:22:49.247545 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 20:22:49.247620 19249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:49.243675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:42377 (local address 127.3.18.1:34605)
0928 20:22:49.243723 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:49.243725 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:49.243850 (+ 125us) server_negotiation.cc:407] Connection header received
0928 20:22:49.243902 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:49.243904 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:49.243931 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:49.243999 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:22:49.244157 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:49.244910 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:49.246507 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:49.247153 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:49.247236 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:49.247330 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:49.247484 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 20:22:49.247546 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 20:22:49.247627 19382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:49.243540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.3:33365 (local address 127.3.18.2:34365)
0928 20:22:49.243609 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:49.243623 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 20:22:49.243685 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:22:49.243939 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:22:49.243940 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:22:49.243942 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:22:49.244033 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:49.244036 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:49.245024 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:49.245025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:49.246402 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:49.246405 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:49.247269 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:49.247275 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:49.247362 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:49.247395 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 20:22:49.247466 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 20:22:49.247530 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":47}
I20260928 20:22:49.247835 19528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:49.243540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.1:34605 (local address 127.3.18.2:42377)
0928 20:22:49.243783 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:49.243792 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 20:22:49.243847 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:22:49.244013 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:22:49.244014 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:22:49.244015 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:22:49.244074 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:49.244076 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:49.245037 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:49.245037 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:49.246402 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:22:49.246405 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:22:49.247296 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:22:49.247297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:22:49.247362 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:49.247395 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 20:22:49.247466 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 20:22:49.247530 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":91,"thread_start_us":112,"threads_started":1}
I20260928 20:22:49.248118 19470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae59f4c4b8344f5582f207387f028714" candidate_uuid: "9891977fbea749aeb0f0e9e80b4f9361" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3638055a7dc94f378d5f16baf99f151d" is_pre_election: true
I20260928 20:22:49.248160 19195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae59f4c4b8344f5582f207387f028714" candidate_uuid: "9891977fbea749aeb0f0e9e80b4f9361" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1073bb7be04e4025bd11239f3303b0e3" is_pre_election: true
I20260928 20:22:49.248333 19195 raft_consensus.cc:2454] T ae59f4c4b8344f5582f207387f028714 P 1073bb7be04e4025bd11239f3303b0e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9891977fbea749aeb0f0e9e80b4f9361 in term 0.
I20260928 20:22:49.248333 19470 raft_consensus.cc:2454] T ae59f4c4b8344f5582f207387f028714 P 3638055a7dc94f378d5f16baf99f151d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9891977fbea749aeb0f0e9e80b4f9361 in term 0.
I20260928 20:22:49.248591 19266 leader_election.cc:302] T ae59f4c4b8344f5582f207387f028714 P 9891977fbea749aeb0f0e9e80b4f9361 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3638055a7dc94f378d5f16baf99f151d, 9891977fbea749aeb0f0e9e80b4f9361; no voters:
I20260928 20:22:49.248781 19385 raft_consensus.cc:2781] T ae59f4c4b8344f5582f207387f028714 P 9891977fbea749aeb0f0e9e80b4f9361 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 20:22:49.248883 19385 raft_consensus.cc:482] T ae59f4c4b8344f5582f207387f028714 P 9891977fbea749aeb0f0e9e80b4f9361 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 20:22:49.248920 19385 raft_consensus.cc:3037] T ae59f4c4b8344f5582f207387f028714 P 9891977fbea749aeb0f0e9e80b4f9361 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:22:49.249667 19385 raft_consensus.cc:504] T ae59f4c4b8344f5582f207387f028714 P 9891977fbea749aeb0f0e9e80b4f9361 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3638055a7dc94f378d5f16baf99f151d" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 33365 } } peers { permanent_uuid: "1073bb7be04e4025bd11239f3303b0e3" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34605 } } peers { permanent_uuid: "9891977fbea749aeb0f0e9e80b4f9361" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 43745 } }
I20260928 20:22:49.249835 19385 leader_election.cc:288] T ae59f4c4b8344f5582f207387f028714 P 9891977fbea749aeb0f0e9e80b4f9361 [CANDIDATE]: Term 1 election: Requested vote from peers 3638055a7dc94f378d5f16baf99f151d (127.3.18.3:33365), 1073bb7be04e4025bd11239f3303b0e3 (127.3.18.1:34605)
I20260928 20:22:49.249985 19195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae59f4c4b8344f5582f207387f028714" candidate_uuid: "9891977fbea749aeb0f0e9e80b4f9361" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1073bb7be04e4025bd11239f3303b0e3"
I20260928 20:22:49.250039 19470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae59f4c4b8344f5582f207387f028714" candidate_uuid: "9891977fbea749aeb0f0e9e80b4f9361" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3638055a7dc94f378d5f16baf99f151d"
I20260928 20:22:49.250087 19195 raft_consensus.cc:3037] T ae59f4c4b8344f5582f207387f028714 P 1073bb7be04e4025bd11239f3303b0e3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:22:49.250126 19470 raft_consensus.cc:3037] T ae59f4c4b8344f5582f207387f028714 P 3638055a7dc94f378d5f16baf99f151d [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:22:49.251020 19470 raft_consensus.cc:2454] T ae59f4c4b8344f5582f207387f028714 P 3638055a7dc94f378d5f16baf99f151d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9891977fbea749aeb0f0e9e80b4f9361 in term 1.
I20260928 20:22:49.251020 19195 raft_consensus.cc:2454] T ae59f4c4b8344f5582f207387f028714 P 1073bb7be04e4025bd11239f3303b0e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9891977fbea749aeb0f0e9e80b4f9361 in term 1.
I20260928 20:22:49.251257 19266 leader_election.cc:302] T ae59f4c4b8344f5582f207387f028714 P 9891977fbea749aeb0f0e9e80b4f9361 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3638055a7dc94f378d5f16baf99f151d, 9891977fbea749aeb0f0e9e80b4f9361; no voters:
I20260928 20:22:49.251355 19385 raft_consensus.cc:2781] T ae59f4c4b8344f5582f207387f028714 P 9891977fbea749aeb0f0e9e80b4f9361 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:22:49.251587 19385 raft_consensus.cc:686] T ae59f4c4b8344f5582f207387f028714 P 9891977fbea749aeb0f0e9e80b4f9361 [term 1 LEADER]: Becoming Leader. State: Replica: 9891977fbea749aeb0f0e9e80b4f9361, State: Running, Role: LEADER
I20260928 20:22:49.251672 19385 consensus_queue.cc:237] T ae59f4c4b8344f5582f207387f028714 P 9891977fbea749aeb0f0e9e80b4f9361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3638055a7dc94f378d5f16baf99f151d" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 33365 } } peers { permanent_uuid: "1073bb7be04e4025bd11239f3303b0e3" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34605 } } peers { permanent_uuid: "9891977fbea749aeb0f0e9e80b4f9361" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 43745 } }
I20260928 20:22:49.252487 19052 catalog_manager.cc:5495] T ae59f4c4b8344f5582f207387f028714 P 9891977fbea749aeb0f0e9e80b4f9361 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9891977fbea749aeb0f0e9e80b4f9361 (127.3.18.2). New cstate: current_term: 1 leader_uuid: "9891977fbea749aeb0f0e9e80b4f9361" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3638055a7dc94f378d5f16baf99f151d" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 33365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1073bb7be04e4025bd11239f3303b0e3" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9891977fbea749aeb0f0e9e80b4f9361" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 43745 } health_report { overall_health: HEALTHY } } }
I20260928 20:22:49.271476 19386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:49.268968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41632 (local address 127.3.18.2:43745)
0928 20:22:49.269032 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 20:22:49.269044 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 20:22:49.269052 (+ 8us) server_negotiation.cc:407] Connection header received
0928 20:22:49.269098 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:22:49.269099 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:22:49.269117 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:22:49.269186 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:22:49.269309 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:49.269885 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:49.270644 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:22:49.270890 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:22:49.270954 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:22:49.271160 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:22:49.271265 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:22:49.271321 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 20:22:49.271397 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 20:22:49.271416 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 20:22:49.273499 19195 raft_consensus.cc:1261] T ae59f4c4b8344f5582f207387f028714 P 1073bb7be04e4025bd11239f3303b0e3 [term 1 FOLLOWER]: Refusing update from remote peer 9891977fbea749aeb0f0e9e80b4f9361: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:22:49.273499 19470 raft_consensus.cc:1261] T ae59f4c4b8344f5582f207387f028714 P 3638055a7dc94f378d5f16baf99f151d [term 1 FOLLOWER]: Refusing update from remote peer 9891977fbea749aeb0f0e9e80b4f9361: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:22:49.273850 19385 consensus_queue.cc:1035] T ae59f4c4b8344f5582f207387f028714 P 9891977fbea749aeb0f0e9e80b4f9361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1073bb7be04e4025bd11239f3303b0e3" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:22:49.274063 19529 consensus_queue.cc:1035] T ae59f4c4b8344f5582f207387f028714 P 9891977fbea749aeb0f0e9e80b4f9361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3638055a7dc94f378d5f16baf99f151d" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 33365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:22:49.276690 19531 mvcc.cc:205] Tried to move back new op lower bound from 7334408066141167616 to 7334408066055487488. Current Snapshot: MvccSnapshot[applied={T|T < 7334408066141167616}]
I20260928 20:22:49.277595 19533 mvcc.cc:205] Tried to move back new op lower bound from 7334408066141167616 to 7334408066055487488. Current Snapshot: MvccSnapshot[applied={T|T < 7334408066141167616}]
I20260928 20:22:49.278964 19051 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:22:49.279088 19051 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:22:49.279910 19051 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb: Sending DeleteTablet for 3 replicas of tablet ae59f4c4b8344f5582f207387f028714
I20260928 20:22:49.280211 19175 tablet_service.cc:1514] Processing DeleteTablet for tablet ae59f4c4b8344f5582f207387f028714 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:22:49 UTC) from {username='kudu', principal='kudu/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:59476
I20260928 20:22:49.280267 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 19090
I20260928 20:22:49.280261 19450 tablet_service.cc:1514] Processing DeleteTablet for tablet ae59f4c4b8344f5582f207387f028714 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:22:49 UTC) from {username='kudu', principal='kudu/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:36864
I20260928 20:22:49.280304 19250 tablet_replica.cc:330] T ae59f4c4b8344f5582f207387f028714 P 1073bb7be04e4025bd11239f3303b0e3: stopping tablet replica
I20260928 20:22:49.280352 19250 raft_consensus.cc:2229] T ae59f4c4b8344f5582f207387f028714 P 1073bb7be04e4025bd11239f3303b0e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:22:49.280395 19250 raft_consensus.cc:2258] T ae59f4c4b8344f5582f207387f028714 P 1073bb7be04e4025bd11239f3303b0e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:22:49.280994 19314 tablet_service.cc:1514] Processing DeleteTablet for tablet ae59f4c4b8344f5582f207387f028714 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:22:49 UTC) from {username='kudu', principal='kudu/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:41628
I20260928 20:22:49.281077 19387 tablet_replica.cc:330] T ae59f4c4b8344f5582f207387f028714 P 9891977fbea749aeb0f0e9e80b4f9361: stopping tablet replica
I20260928 20:22:49.281088 19540 tablet_replica.cc:330] T ae59f4c4b8344f5582f207387f028714 P 3638055a7dc94f378d5f16baf99f151d: stopping tablet replica
I20260928 20:22:49.281124 19387 raft_consensus.cc:2229] T ae59f4c4b8344f5582f207387f028714 P 9891977fbea749aeb0f0e9e80b4f9361 [term 1 LEADER]: Raft consensus shutting down.
I20260928 20:22:49.281196 19540 raft_consensus.cc:2229] T ae59f4c4b8344f5582f207387f028714 P 3638055a7dc94f378d5f16baf99f151d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:22:49.281211 19387 raft_consensus.cc:2258] T ae59f4c4b8344f5582f207387f028714 P 9891977fbea749aeb0f0e9e80b4f9361 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:22:49.281311 19540 raft_consensus.cc:2258] T ae59f4c4b8344f5582f207387f028714 P 3638055a7dc94f378d5f16baf99f151d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:22:49.281512 19387 ts_tablet_manager.cc:1905] T ae59f4c4b8344f5582f207387f028714 P 9891977fbea749aeb0f0e9e80b4f9361: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:22:49.283018 19540 ts_tablet_manager.cc:1905] T ae59f4c4b8344f5582f207387f028714 P 3638055a7dc94f378d5f16baf99f151d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:22:49.283273 19387 ts_tablet_manager.cc:1918] T ae59f4c4b8344f5582f207387f028714 P 9891977fbea749aeb0f0e9e80b4f9361: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:22:49.283365 19387 log.cc:1196] T ae59f4c4b8344f5582f207387f028714 P 9891977fbea749aeb0f0e9e80b4f9361: Deleting WAL directory at /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-1/wal/wals/ae59f4c4b8344f5582f207387f028714
I20260928 20:22:49.283632 19387 ts_tablet_manager.cc:1939] T ae59f4c4b8344f5582f207387f028714 P 9891977fbea749aeb0f0e9e80b4f9361: Deleting consensus metadata
I20260928 20:22:49.284025 19036 catalog_manager.cc:4841] TS 9891977fbea749aeb0f0e9e80b4f9361 (127.3.18.2:43745): tablet ae59f4c4b8344f5582f207387f028714 (table test-table [id=b4bc838b59e2441998b4667af4cb3e1d]) successfully deleted
I20260928 20:22:49.284956 19540 ts_tablet_manager.cc:1918] T ae59f4c4b8344f5582f207387f028714 P 3638055a7dc94f378d5f16baf99f151d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:22:49.285040 19540 log.cc:1196] T ae59f4c4b8344f5582f207387f028714 P 3638055a7dc94f378d5f16baf99f151d: Deleting WAL directory at /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-2/wal/wals/ae59f4c4b8344f5582f207387f028714
I20260928 20:22:49.285331 19540 ts_tablet_manager.cc:1939] T ae59f4c4b8344f5582f207387f028714 P 3638055a7dc94f378d5f16baf99f151d: Deleting consensus metadata
I20260928 20:22:49.285787 19036 catalog_manager.cc:4841] TS 3638055a7dc94f378d5f16baf99f151d (127.3.18.3:33365): tablet ae59f4c4b8344f5582f207387f028714 (table test-table [id=b4bc838b59e2441998b4667af4cb3e1d]) successfully deleted
W20260928 20:22:49.286861 19037 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.18.1:34605 (error 108)
I20260928 20:22:49.287261 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 19246
I20260928 20:22:49.288193 19248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:22:49.287399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.1:34605 (local address 127.0.0.1:59478)
0928 20:22:49.288103 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0928 20:22:49.288151 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.1:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":682}
W20260928 20:22:49.288332 19037 catalog_manager.cc:4584] TS 1073bb7be04e4025bd11239f3303b0e3 (127.3.18.1:34605): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ae59f4c4b8344f5582f207387f028714: Network error: Client connection negotiation failed: client connection to 127.3.18.1:34605: connect: Connection refused (error 111)
I20260928 20:22:49.292708 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 19384
I20260928 20:22:49.297554 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 19019
I20260928 20:23:09.303366 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:40865
--webserver_interface=127.3.18.62
--webserver_port=45599
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:36695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.18.62:40865
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:23:09.386871 19542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:23:09.387100 19542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:23:09.387123 19542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:23:09.388381 19542 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 20:23:09.388427 19542 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 20:23:09.388441 19542 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 20:23:09.388453 19542 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:23:09.388476 19542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:23:09.388490 19542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:23:09.388504 19542 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:23:09.388515 19542 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:23:09.390062 19542 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:36695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.18.62:40865
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:40865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.18.62
--webserver_port=45599
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:23:09.390293 19542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:23:09.390479 19542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:23:09.396023 19547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:23:09.396049 19542 server_base.cc:1034] running on GCE node
W20260928 20:23:09.396026 19550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:09.396041 19548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:23:09.396534 19542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:23:09.396904 19542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:23:09.398087 19542 hybrid_clock.cc:648] HybridClock initialized: now 1790626989398084 us; error 36 us; skew 500 ppm
Sep 28 20:23:09 dist-test-slave-b83v krb5kdc[18472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626989, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:23:09.406214 19542 init.cc:377] Logged in from keytab as kudu/127.3.18.62@KRBTEST.COM (short username kudu)
I20260928 20:23:09.406772 19542 webserver.cc:460] Webserver started at http://127.3.18.62:45599/ using document root <none> and password file <none>
I20260928 20:23:09.407047 19542 fs_manager.cc:359] Metadata directory not provided
I20260928 20:23:09.407124 19542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:23:09.408722 19542 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:23:09.409493 19557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:09.409816 19542 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:23:09.409926 19542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/master-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/master-0/wal
uuid: "fe5679c7c74f4b3e843599dc347a86bb"
format_stamp: "Formatted at 2026-09-28 20:22:27 on dist-test-slave-b83v"
server_key: "3c5671d9e419d5c04f18d0b2d10192a3"
server_key_iv: "dd70f2bec837b6baa8d55db353a06a97"
server_key_version: "encryptionkey@0"
I20260928 20:23:09.410286 19542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:23:09.430008 19542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:23:09.432484 19542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:23:09.432662 19542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:23:09.436781 19542 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.62:40865
I20260928 20:23:09.436914 19609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.62:40865 every 8 connection(s)
I20260928 20:23:09.437284 19542 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
I20260928 20:23:09.438755 19610 sys_catalog.cc:263] Verifying existing consensus state
I20260928 20:23:09.438982 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 19542
I20260928 20:23:09.439283 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.1:34605
--local_ip_for_outbound_sockets=127.3.18.1
--tserver_master_addrs=127.3.18.62:40865
--webserver_port=41063
--webserver_interface=127.3.18.1
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab
--principal=kudu/127.3.18.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:36695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 20:23:09.439886 19610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb: Bootstrap starting.
I20260928 20:23:09.448941 19610 log.cc:824] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb: Log is configured to *not* fsync() on all Append() calls
I20260928 20:23:09.451304 19610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 20:23:09.451638 19610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb: Bootstrap complete.
I20260928 20:23:09.454030 19610 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe5679c7c74f4b3e843599dc347a86bb" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 40865 } }
I20260928 20:23:09.454268 19610 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe5679c7c74f4b3e843599dc347a86bb, State: Initialized, Role: FOLLOWER
I20260928 20:23:09.454473 19610 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe5679c7c74f4b3e843599dc347a86bb" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 40865 } }
I20260928 20:23:09.454535 19610 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:23:09.454609 19610 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:23:09.454656 19610 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb [term 2 FOLLOWER]: Advancing to term 3
I20260928 20:23:09.455680 19610 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe5679c7c74f4b3e843599dc347a86bb" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 40865 } }
I20260928 20:23:09.455811 19610 leader_election.cc:302] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe5679c7c74f4b3e843599dc347a86bb; no voters:
I20260928 20:23:09.456022 19610 leader_election.cc:288] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 20:23:09.456058 19614 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb [term 3 FOLLOWER]: Leader election won for term 3
I20260928 20:23:09.456171 19614 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb [term 3 LEADER]: Becoming Leader. State: Replica: fe5679c7c74f4b3e843599dc347a86bb, State: Running, Role: LEADER
I20260928 20:23:09.456308 19610 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:23:09.456331 19614 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe5679c7c74f4b3e843599dc347a86bb" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 40865 } }
I20260928 20:23:09.456620 19614 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe5679c7c74f4b3e843599dc347a86bb. Latest consensus state: current_term: 3 leader_uuid: "fe5679c7c74f4b3e843599dc347a86bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe5679c7c74f4b3e843599dc347a86bb" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 40865 } } }
I20260928 20:23:09.456625 19615 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "fe5679c7c74f4b3e843599dc347a86bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe5679c7c74f4b3e843599dc347a86bb" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 40865 } } }
I20260928 20:23:09.456775 19615 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb [sys.catalog]: This master's current role is: LEADER
I20260928 20:23:09.456754 19614 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb [sys.catalog]: This master's current role is: LEADER
I20260928 20:23:09.457335 19624 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:23:09.458148 19624 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:23:09.458261 19624 catalog_manager.cc:1259] Loaded cluster ID: a6286dddff7642f1866ae832f5093957
I20260928 20:23:09.458453 19624 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:23:09.459719 19624 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:23:09.459932 19624 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb: Loaded TSK: 0
I20260928 20:23:09.459995 19624 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fe5679c7c74f4b3e843599dc347a86bb: Loaded TSK: 1
F20260928 20:23:09.460212 19624 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f80db58c9ad google::LogMessage::Fail() at ??:0
@ 0x7f80db58f88a google::LogMessage::SendToLog() at ??:0
@ 0x7f80db58c4de google::LogMessage::Flush() at ??:0
@ 0x7f80db58da19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f80ddcf9104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f80ddcf6d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f80e34ea1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f80e34ea48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f80e34ea594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f80e34b0c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f80e34baad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f80dd45a0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f80dd44f954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f80de2b36db start_thread at ??:0
@ 0x7f80da08a61f clone at ??:0
W20260928 20:23:09.541893 19612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:23:09.542153 19612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:23:09.542201 19612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:23:09.543453 19612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:23:09.543531 19612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:23:09.543579 19612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.1
I20260928 20:23:09.545364 19612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:36695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab
--principal=kudu/127.3.18.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.1:34605
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.18.1
--webserver_port=41063
--tserver_master_addrs=127.3.18.62:40865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.1
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:23:09.545674 19612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:23:09.545928 19612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:23:09.551411 19651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:09.551411 19652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:23:09.551739 19612 server_base.cc:1034] running on GCE node
W20260928 20:23:09.551822 19654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:23:09.552033 19612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:23:09.552341 19612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:23:09.553531 19612 hybrid_clock.cc:648] HybridClock initialized: now 1790626989553519 us; error 28 us; skew 500 ppm
Sep 28 20:23:09 dist-test-slave-b83v krb5kdc[18472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626989, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:23:09.556598 19612 init.cc:377] Logged in from keytab as kudu/127.3.18.1@KRBTEST.COM (short username kudu)
I20260928 20:23:09.557211 19612 webserver.cc:460] Webserver started at http://127.3.18.1:41063/ using document root <none> and password file <none>
I20260928 20:23:09.557525 19612 fs_manager.cc:359] Metadata directory not provided
I20260928 20:23:09.557605 19612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:23:09.559473 19612 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 20:23:09.560523 19661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:09.560870 19612 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:23:09.560982 19612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-0/wal
uuid: "1073bb7be04e4025bd11239f3303b0e3"
format_stamp: "Formatted at 2026-09-28 20:22:28 on dist-test-slave-b83v"
server_key: "5f970228296b9f459911d0b49a1c54d3"
server_key_iv: "290dac0c00c973deaf9703386fdfbd18"
server_key_version: "encryptionkey@0"
I20260928 20:23:09.561395 19612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:23:09.570041 19612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:23:09.573323 19612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:23:09.573545 19612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:23:09.573855 19612 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:23:09.574402 19668 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 20:23:09.576591 19612 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 20:23:09.576764 19612 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20260928 20:23:09.576855 19612 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 20:23:09.577653 19612 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 20:23:09.577718 19612 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 20:23:09.577802 19668 tablet_bootstrap.cc:492] T ae59f4c4b8344f5582f207387f028714 P 1073bb7be04e4025bd11239f3303b0e3: Bootstrap starting.
I20260928 20:23:09.586105 19612 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.1:34605
I20260928 20:23:09.586284 19775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.1:34605 every 8 connection(s)
I20260928 20:23:09.586606 19612 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
I20260928 20:23:09.589602 19668 log.cc:824] T ae59f4c4b8344f5582f207387f028714 P 1073bb7be04e4025bd11239f3303b0e3: Log is configured to *not* fsync() on all Append() calls
I20260928 20:23:09.592790 19668 tablet_bootstrap.cc:492] T ae59f4c4b8344f5582f207387f028714 P 1073bb7be04e4025bd11239f3303b0e3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 20:23:09.593150 19668 tablet_bootstrap.cc:492] T ae59f4c4b8344f5582f207387f028714 P 1073bb7be04e4025bd11239f3303b0e3: Bootstrap complete.
I20260928 20:23:09.594424 19668 ts_tablet_manager.cc:1397] T ae59f4c4b8344f5582f207387f028714 P 1073bb7be04e4025bd11239f3303b0e3: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.011s
I20260928 20:23:09.595346 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 19612
I20260928 20:23:09.595651 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.2:43745
--local_ip_for_outbound_sockets=127.3.18.2
--tserver_master_addrs=127.3.18.62:40865
--webserver_port=44901
--webserver_interface=127.3.18.2
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab
--principal=kudu/127.3.18.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:36695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 20:23:09.596064 19668 raft_consensus.cc:348] T ae59f4c4b8344f5582f207387f028714 P 1073bb7be04e4025bd11239f3303b0e3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3638055a7dc94f378d5f16baf99f151d" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 33365 } } peers { permanent_uuid: "1073bb7be04e4025bd11239f3303b0e3" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34605 } } peers { permanent_uuid: "9891977fbea749aeb0f0e9e80b4f9361" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 43745 } }
I20260928 20:23:09.596426 19668 raft_consensus.cc:729] T ae59f4c4b8344f5582f207387f028714 P 1073bb7be04e4025bd11239f3303b0e3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1073bb7be04e4025bd11239f3303b0e3, State: Initialized, Role: FOLLOWER
I20260928 20:23:09.596596 19668 consensus_queue.cc:260] T ae59f4c4b8344f5582f207387f028714 P 1073bb7be04e4025bd11239f3303b0e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3638055a7dc94f378d5f16baf99f151d" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 33365 } } peers { permanent_uuid: "1073bb7be04e4025bd11239f3303b0e3" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34605 } } peers { permanent_uuid: "9891977fbea749aeb0f0e9e80b4f9361" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 43745 } }
I20260928 20:23:09.596903 19668 ts_tablet_manager.cc:1428] T ae59f4c4b8344f5582f207387f028714 P 1073bb7be04e4025bd11239f3303b0e3: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
W20260928 20:23:09.674393 19778 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:23:09.587498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40865 (local address 127.3.18.1:44747)
0928 20:23:09.587705 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:09.587788 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0928 20:23:09.588377 (+ 589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:23:09.674166 (+ 85789us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:40865: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":116,"threads_started":1}
W20260928 20:23:09.674731 19776 heartbeater.cc:644] Failed to heartbeat to 127.3.18.62:40865 (0 consecutive failures): Network error: Failed to ping master at 127.3.18.62:40865: Client connection negotiation failed: client connection to 127.3.18.62:40865: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 20:23:09.675140 19778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:09.675007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40865 (local address 127.3.18.1:34439)
0928 20:23:09.675067 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:09.675112 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:40865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 20:23:09.675503 19778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:09.675409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40865 (local address 127.3.18.1:59261)
0928 20:23:09.675454 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:09.675482 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:40865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 20:23:09.675688 19776 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 20:23:09.680102 19781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:23:09.680335 19781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:23:09.680361 19781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:23:09.681594 19781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:23:09.681636 19781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:23:09.681666 19781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.2
I20260928 20:23:09.683213 19781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:36695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab
--principal=kudu/127.3.18.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.2:43745
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.18.2
--webserver_port=44901
--tserver_master_addrs=127.3.18.62:40865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19781
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.2
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:23:09.683475 19781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:23:09.683714 19781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 20:23:09.689057 19781 server_base.cc:1034] running on GCE node
W20260928 20:23:09.689028 19786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:09.689051 19789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:09.689029 19787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:23:09.689409 19781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:23:09.689612 19781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:23:09.690841 19781 hybrid_clock.cc:648] HybridClock initialized: now 1790626989690802 us; error 45 us; skew 500 ppm
Sep 28 20:23:09 dist-test-slave-b83v krb5kdc[18472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626989, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:23:09.693596 19781 init.cc:377] Logged in from keytab as kudu/127.3.18.2@KRBTEST.COM (short username kudu)
I20260928 20:23:09.694116 19781 webserver.cc:460] Webserver started at http://127.3.18.2:44901/ using document root <none> and password file <none>
I20260928 20:23:09.694373 19781 fs_manager.cc:359] Metadata directory not provided
I20260928 20:23:09.694422 19781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:23:09.695995 19781 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:23:09.696636 19796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:09.696813 19781 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:09.696916 19781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-1/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-1/wal
uuid: "9891977fbea749aeb0f0e9e80b4f9361"
format_stamp: "Formatted at 2026-09-28 20:22:28 on dist-test-slave-b83v"
server_key: "0c19841604e3953e1aedad39d14244a9"
server_key_iv: "3300c18e4170e2c69334ff0566b0733a"
server_key_version: "encryptionkey@0"
I20260928 20:23:09.697281 19781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:23:09.710827 19781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:23:09.713110 19781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:23:09.713322 19781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:23:09.713585 19781 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:23:09.713935 19781 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:23:09.713990 19781 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:09.714035 19781 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:23:09.714073 19781 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:09.720705 19781 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.2:43745
I20260928 20:23:09.720752 19909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.2:43745 every 8 connection(s)
I20260928 20:23:09.721117 19781 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
I20260928 20:23:09.721674 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 19781
I20260928 20:23:09.721956 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.3:33365
--local_ip_for_outbound_sockets=127.3.18.3
--tserver_master_addrs=127.3.18.62:40865
--webserver_port=38473
--webserver_interface=127.3.18.3
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab
--principal=kudu/127.3.18.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:36695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 20:23:09.722520 19912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:09.721938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40865 (local address 127.3.18.2:37265)
0928 20:23:09.722217 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:09.722311 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:40865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":93,"threads_started":1}
W20260928 20:23:09.723054 19910 heartbeater.cc:644] Failed to heartbeat to 127.3.18.62:40865 (0 consecutive failures): Network error: Failed to ping master at 127.3.18.62:40865: Client connection negotiation failed: client connection to 127.3.18.62:40865: connect: Connection refused (error 111)
I20260928 20:23:09.723398 19912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:09.723290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40865 (local address 127.3.18.2:50391)
0928 20:23:09.723344 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:09.723375 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:40865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 20:23:09.723718 19912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:09.723593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40865 (local address 127.3.18.2:59749)
0928 20:23:09.723638 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:09.723701 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:40865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 20:23:09.723842 19910 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 20:23:09.806119 19913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:23:09.806327 19913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:23:09.806353 19913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:23:09.807633 19913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:23:09.807677 19913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:23:09.807703 19913 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.3
I20260928 20:23:09.809217 19913 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:36695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab
--principal=kudu/127.3.18.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.3:33365
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.18.3
--webserver_port=38473
--tserver_master_addrs=127.3.18.62:40865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19913
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.3
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:23:09.809461 19913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:23:09.809715 19913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:23:09.815120 19921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:09.815136 19919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:09.815136 19918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:23:09.815173 19913 server_base.cc:1034] running on GCE node
I20260928 20:23:09.815593 19913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:23:09.815892 19913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:23:09.817066 19913 hybrid_clock.cc:648] HybridClock initialized: now 1790626989817057 us; error 28 us; skew 500 ppm
Sep 28 20:23:09 dist-test-slave-b83v krb5kdc[18472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626989, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:23:09.819644 19913 init.cc:377] Logged in from keytab as kudu/127.3.18.3@KRBTEST.COM (short username kudu)
I20260928 20:23:09.820118 19913 webserver.cc:460] Webserver started at http://127.3.18.3:38473/ using document root <none> and password file <none>
I20260928 20:23:09.820354 19913 fs_manager.cc:359] Metadata directory not provided
I20260928 20:23:09.820425 19913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:23:09.821883 19913 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 20:23:09.822580 19928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:09.822918 19913 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:23:09.823025 19913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-2/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-2/wal
uuid: "3638055a7dc94f378d5f16baf99f151d"
format_stamp: "Formatted at 2026-09-28 20:22:28 on dist-test-slave-b83v"
server_key: "03c355bf2c34de2ec07ddabb3293d980"
server_key_iv: "c64512be4d15bbebfff90112b37c8c6b"
server_key_version: "encryptionkey@0"
I20260928 20:23:09.823398 19913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:23:09.846424 19913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:23:09.849025 19913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:23:09.849221 19913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:23:09.849499 19913 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:23:09.849845 19913 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:23:09.849901 19913 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:09.849937 19913 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:23:09.849983 19913 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:09.856072 19913 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.3:33365
I20260928 20:23:09.856153 20041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.3:33365 every 8 connection(s)
I20260928 20:23:09.856695 19913 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
I20260928 20:23:09.857745 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 19913
I20260928 20:23:09.858371 20044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:09.857815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40865 (local address 127.3.18.3:35205)
0928 20:23:09.858029 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:09.858135 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:40865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":105,"threads_started":1}
W20260928 20:23:09.858840 20042 heartbeater.cc:644] Failed to heartbeat to 127.3.18.62:40865 (0 consecutive failures): Network error: Failed to ping master at 127.3.18.62:40865: Client connection negotiation failed: client connection to 127.3.18.62:40865: connect: Connection refused (error 111)
I20260928 20:23:09.859193 20044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:09.859082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40865 (local address 127.3.18.3:57647)
0928 20:23:09.859139 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:09.859167 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:40865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 20:23:09.859527 20044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:09.859429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40865 (local address 127.3.18.3:32819)
0928 20:23:09.859488 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:09.859511 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:40865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 20:23:09.859736 20042 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 20:23:10.676410 20046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:10.676145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40865 (local address 127.3.18.1:55687)
0928 20:23:10.676297 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:10.676345 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:40865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":69,"threads_started":1}
I20260928 20:23:10.724593 20047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:10.724313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40865 (local address 127.3.18.2:39155)
0928 20:23:10.724448 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:10.724506 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:40865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":67,"threads_started":1}
I20260928 20:23:10.860389 20048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:10.860103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40865 (local address 127.3.18.3:53501)
0928 20:23:10.860283 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:10.860330 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:40865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":69,"threads_started":1}
I20260928 20:23:10.950750 20049 raft_consensus.cc:482] T ae59f4c4b8344f5582f207387f028714 P 1073bb7be04e4025bd11239f3303b0e3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:23:10.950877 20049 raft_consensus.cc:504] T ae59f4c4b8344f5582f207387f028714 P 1073bb7be04e4025bd11239f3303b0e3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3638055a7dc94f378d5f16baf99f151d" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 33365 } } peers { permanent_uuid: "1073bb7be04e4025bd11239f3303b0e3" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34605 } } peers { permanent_uuid: "9891977fbea749aeb0f0e9e80b4f9361" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 43745 } }
I20260928 20:23:10.951186 20049 leader_election.cc:288] T ae59f4c4b8344f5582f207387f028714 P 1073bb7be04e4025bd11239f3303b0e3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3638055a7dc94f378d5f16baf99f151d (127.3.18.3:33365), 9891977fbea749aeb0f0e9e80b4f9361 (127.3.18.2:43745)
Sep 28 20:23:10 dist-test-slave-b83v krb5kdc[18472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626989, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.1@KRBTEST.COM for kudu/127.3.18.2@KRBTEST.COM
Sep 28 20:23:10 dist-test-slave-b83v krb5kdc[18472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790626989, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.1@KRBTEST.COM for kudu/127.3.18.3@KRBTEST.COM
I20260928 20:23:10.965229 20051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:10.951616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.2:43745 (local address 127.3.18.1:40057)
0928 20:23:10.951850 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:10.951862 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:23:10.951909 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:23:10.952657 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:23:10.952660 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:23:10.953288 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:23:10.953555 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:10.953558 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:10.955287 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:10.955289 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:10.956734 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:10.956736 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:10.957699 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:10.957700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:10.957767 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:10.958488 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:23:10.958493 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:23:10.960552 (+ 2059us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:23:10.963322 (+ 2770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:23:10.963333 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:23:10.963336 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:23:10.963569 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:23:10.963905 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:23:10.963906 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:23:10.963907 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:23:10.964051 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:23:10.964598 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:23:10.964605 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:23:10.964779 (+ 174us) client_negotiation.cc:769] Sending connection context
0928 20:23:10.964882 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 20:23:10.965082 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":89,"thread_start_us":180,"threads_started":1}
I20260928 20:23:10.965337 20050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:10.951628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:33293 (local address 127.3.18.3:33365)
0928 20:23:10.951917 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:10.951928 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:10.951962 (+ 34us) server_negotiation.cc:407] Connection header received
0928 20:23:10.952405 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:10.952408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:10.952448 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:10.952619 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:10.953695 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:10.955201 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:10.956843 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:10.957432 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:10.957583 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:10.961365 (+ 3782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:10.961369 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:10.961369 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:23:10.961384 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:10.963636 (+ 2252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:10.964066 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:10.964069 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:10.964069 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:10.964181 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:10.964425 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:10.964426 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:10.964426 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:10.964808 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:10.964894 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:10.965081 (+ 187us) server_negotiation.cc:299] Negotiation successful
0928 20:23:10.965213 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":88,"threads_started":1}
I20260928 20:23:10.965518 20052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:10.951728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:40057 (local address 127.3.18.2:43745)
0928 20:23:10.951882 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:10.951894 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:10.951929 (+ 35us) server_negotiation.cc:407] Connection header received
0928 20:23:10.952395 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:10.952397 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:10.952442 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:10.952619 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:10.953706 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:10.955154 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:10.956843 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:10.957451 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:10.957579 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:10.960708 (+ 3129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:10.960713 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:10.960714 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:23:10.960727 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:10.963224 (+ 2497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:10.963678 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:10.963682 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:10.963682 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:10.963827 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:10.964153 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:10.964155 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:10.964155 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:10.964543 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:10.964609 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:10.964961 (+ 352us) server_negotiation.cc:299] Negotiation successful
0928 20:23:10.965155 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":80,"threads_started":1}
I20260928 20:23:10.965315 20046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:10.951460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.3:33365 (local address 127.3.18.1:33293)
0928 20:23:10.951598 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:10.951614 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 20:23:10.951746 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:23:10.952657 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:23:10.952660 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:23:10.953244 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:23:10.953540 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:10.953550 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:10.955313 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:10.955314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:10.956738 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:10.956741 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:10.957562 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:10.957563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:10.957686 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:10.958430 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:23:10.958443 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:23:10.961224 (+ 2781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:23:10.963724 (+ 2500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:23:10.963725 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:23:10.963725 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:23:10.963974 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:23:10.964244 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:23:10.964245 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:23:10.964245 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:23:10.964320 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:23:10.964886 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:23:10.964888 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:23:10.964939 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 20:23:10.965004 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 20:23:10.965156 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":111}
W20260928 20:23:10.966418 19662 leader_election.cc:341] T ae59f4c4b8344f5582f207387f028714 P 1073bb7be04e4025bd11239f3303b0e3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9891977fbea749aeb0f0e9e80b4f9361 (127.3.18.2:43745): Not found: Tablet not found: ae59f4c4b8344f5582f207387f028714
W20260928 20:23:10.966523 19662 leader_election.cc:341] T ae59f4c4b8344f5582f207387f028714 P 1073bb7be04e4025bd11239f3303b0e3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3638055a7dc94f378d5f16baf99f151d (127.3.18.3:33365): Not found: Tablet not found: ae59f4c4b8344f5582f207387f028714
I20260928 20:23:10.966581 19662 leader_election.cc:302] T ae59f4c4b8344f5582f207387f028714 P 1073bb7be04e4025bd11239f3303b0e3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1073bb7be04e4025bd11239f3303b0e3; no voters: 3638055a7dc94f378d5f16baf99f151d, 9891977fbea749aeb0f0e9e80b4f9361
I20260928 20:23:10.966710 20049 raft_consensus.cc:2726] T ae59f4c4b8344f5582f207387f028714 P 1073bb7be04e4025bd11239f3303b0e3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 20:23:11.677235 20053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:11.676945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40865 (local address 127.3.18.1:33933)
0928 20:23:11.677111 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:11.677162 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:40865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":67,"threads_started":1}
I20260928 20:23:11.725458 20054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:11.725167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40865 (local address 127.3.18.2:56791)
0928 20:23:11.725345 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:11.725384 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:40865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":51,"threads_started":1}
I20260928 20:23:11.861325 20055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:11.860995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40865 (local address 127.3.18.3:42693)
0928 20:23:11.861226 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:11.861275 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:40865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":99,"threads_started":1}
I20260928 20:23:12.678071 20056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:12.677826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40865 (local address 127.3.18.1:60985)
0928 20:23:12.677977 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:12.678021 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:40865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":66,"threads_started":1}
I20260928 20:23:12.726227 20057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:12.725926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40865 (local address 127.3.18.2:45915)
0928 20:23:12.726072 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:12.726119 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:40865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":76,"threads_started":1}
I20260928 20:23:12.862313 20058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:12.861994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40865 (local address 127.3.18.3:59519)
0928 20:23:12.862209 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:12.862264 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:40865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":67,"threads_started":1}
I20260928 20:23:13.056365 20059 raft_consensus.cc:482] T ae59f4c4b8344f5582f207387f028714 P 1073bb7be04e4025bd11239f3303b0e3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:23:13.056473 20059 raft_consensus.cc:504] T ae59f4c4b8344f5582f207387f028714 P 1073bb7be04e4025bd11239f3303b0e3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3638055a7dc94f378d5f16baf99f151d" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 33365 } } peers { permanent_uuid: "1073bb7be04e4025bd11239f3303b0e3" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34605 } } peers { permanent_uuid: "9891977fbea749aeb0f0e9e80b4f9361" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 43745 } }
I20260928 20:23:13.056613 20059 leader_election.cc:288] T ae59f4c4b8344f5582f207387f028714 P 1073bb7be04e4025bd11239f3303b0e3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3638055a7dc94f378d5f16baf99f151d (127.3.18.3:33365), 9891977fbea749aeb0f0e9e80b4f9361 (127.3.18.2:43745)
W20260928 20:23:13.057224 19662 leader_election.cc:341] T ae59f4c4b8344f5582f207387f028714 P 1073bb7be04e4025bd11239f3303b0e3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3638055a7dc94f378d5f16baf99f151d (127.3.18.3:33365): Not found: Tablet not found: ae59f4c4b8344f5582f207387f028714
W20260928 20:23:13.057324 19662 leader_election.cc:341] T ae59f4c4b8344f5582f207387f028714 P 1073bb7be04e4025bd11239f3303b0e3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9891977fbea749aeb0f0e9e80b4f9361 (127.3.18.2:43745): Not found: Tablet not found: ae59f4c4b8344f5582f207387f028714
I20260928 20:23:13.057356 19662 leader_election.cc:302] T ae59f4c4b8344f5582f207387f028714 P 1073bb7be04e4025bd11239f3303b0e3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1073bb7be04e4025bd11239f3303b0e3; no voters: 3638055a7dc94f378d5f16baf99f151d, 9891977fbea749aeb0f0e9e80b4f9361
I20260928 20:23:13.057459 20059 raft_consensus.cc:2726] T ae59f4c4b8344f5582f207387f028714 P 1073bb7be04e4025bd11239f3303b0e3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 20:23:13.678903 20060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:13.678631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40865 (local address 127.3.18.1:52905)
0928 20:23:13.678807 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:13.678856 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:40865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":67,"threads_started":1}
I20260928 20:23:13.727214 20061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:13.726869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40865 (local address 127.3.18.2:46113)
0928 20:23:13.727094 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:13.727162 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:40865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":96,"threads_started":1}
I20260928 20:23:13.863343 20062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:13.863076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40865 (local address 127.3.18.3:35131)
0928 20:23:13.863245 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:13.863293 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:40865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":73,"threads_started":1}
I20260928 20:23:14.622939 20063 raft_consensus.cc:482] T ae59f4c4b8344f5582f207387f028714 P 1073bb7be04e4025bd11239f3303b0e3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:23:14.623035 20063 raft_consensus.cc:504] T ae59f4c4b8344f5582f207387f028714 P 1073bb7be04e4025bd11239f3303b0e3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3638055a7dc94f378d5f16baf99f151d" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 33365 } } peers { permanent_uuid: "1073bb7be04e4025bd11239f3303b0e3" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34605 } } peers { permanent_uuid: "9891977fbea749aeb0f0e9e80b4f9361" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 43745 } }
I20260928 20:23:14.623353 20063 leader_election.cc:288] T ae59f4c4b8344f5582f207387f028714 P 1073bb7be04e4025bd11239f3303b0e3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3638055a7dc94f378d5f16baf99f151d (127.3.18.3:33365), 9891977fbea749aeb0f0e9e80b4f9361 (127.3.18.2:43745)
W20260928 20:23:14.623895 19662 leader_election.cc:341] T ae59f4c4b8344f5582f207387f028714 P 1073bb7be04e4025bd11239f3303b0e3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3638055a7dc94f378d5f16baf99f151d (127.3.18.3:33365): Not found: Tablet not found: ae59f4c4b8344f5582f207387f028714
W20260928 20:23:14.623989 19662 leader_election.cc:341] T ae59f4c4b8344f5582f207387f028714 P 1073bb7be04e4025bd11239f3303b0e3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9891977fbea749aeb0f0e9e80b4f9361 (127.3.18.2:43745): Not found: Tablet not found: ae59f4c4b8344f5582f207387f028714
I20260928 20:23:14.624024 19662 leader_election.cc:302] T ae59f4c4b8344f5582f207387f028714 P 1073bb7be04e4025bd11239f3303b0e3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1073bb7be04e4025bd11239f3303b0e3; no voters: 3638055a7dc94f378d5f16baf99f151d, 9891977fbea749aeb0f0e9e80b4f9361
I20260928 20:23:14.624152 20063 raft_consensus.cc:2726] T ae59f4c4b8344f5582f207387f028714 P 1073bb7be04e4025bd11239f3303b0e3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 20:23:14.679702 20064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:14.679435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40865 (local address 127.3.18.1:59941)
0928 20:23:14.679598 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:14.679653 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:40865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":66,"threads_started":1}
I20260928 20:23:14.728050 20065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:14.727817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40865 (local address 127.3.18.2:59849)
0928 20:23:14.727964 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:14.728004 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:40865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":81,"threads_started":1}
I20260928 20:23:14.864336 20066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:14.864040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40865 (local address 127.3.18.3:44749)
0928 20:23:14.864239 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:14.864285 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:40865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":115,"threads_started":1}
I20260928 20:23:15.680794 20067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:15.680526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40865 (local address 127.3.18.1:35883)
0928 20:23:15.680676 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:15.680743 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:40865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":69,"threads_started":1}
I20260928 20:23:15.728907 20068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:15.728642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40865 (local address 127.3.18.2:48685)
0928 20:23:15.728810 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:15.728854 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:40865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":76,"threads_started":1}
I20260928 20:23:15.865051 20069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:15.864844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40865 (local address 127.3.18.3:37335)
0928 20:23:15.864964 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:15.864999 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:40865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":62,"threads_started":1}
I20260928 20:23:16.681734 20070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:16.681441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40865 (local address 127.3.18.1:51825)
0928 20:23:16.681637 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:16.681684 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:40865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":87,"threads_started":1}
I20260928 20:23:16.729787 20071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:16.729538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40865 (local address 127.3.18.2:48039)
0928 20:23:16.729705 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:16.729740 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:40865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":74,"threads_started":1}
I20260928 20:23:16.865934 20072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:16.865587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40865 (local address 127.3.18.3:56997)
0928 20:23:16.865841 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:16.865886 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:40865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":131,"threads_started":1}
I20260928 20:23:17.682516 20073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:17.682288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40865 (local address 127.3.18.1:42957)
0928 20:23:17.682427 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:17.682469 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:40865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":79,"threads_started":1}
I20260928 20:23:17.730999 20074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:17.730601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40865 (local address 127.3.18.2:41565)
0928 20:23:17.730874 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:17.730940 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:40865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":124,"threads_started":1}
I20260928 20:23:17.866894 20075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:17.866585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40865 (local address 127.3.18.3:34903)
0928 20:23:17.866786 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:17.866833 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:40865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":127,"threads_started":1}
I20260928 20:23:18.683305 20076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:18.683043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40865 (local address 127.3.18.1:43567)
0928 20:23:18.683212 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:18.683255 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:40865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":64,"threads_started":1}
I20260928 20:23:18.732069 20077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:18.731716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40865 (local address 127.3.18.2:59187)
0928 20:23:18.731964 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:18.732013 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:40865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":168,"threads_started":1}
I20260928 20:23:18.867710 20078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:18.867441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40865 (local address 127.3.18.3:46957)
0928 20:23:18.867616 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:18.867662 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:40865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":87,"threads_started":1}
I20260928 20:23:19.511888 20079 raft_consensus.cc:482] T ae59f4c4b8344f5582f207387f028714 P 1073bb7be04e4025bd11239f3303b0e3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:23:19.512012 20079 raft_consensus.cc:504] T ae59f4c4b8344f5582f207387f028714 P 1073bb7be04e4025bd11239f3303b0e3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3638055a7dc94f378d5f16baf99f151d" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 33365 } } peers { permanent_uuid: "1073bb7be04e4025bd11239f3303b0e3" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34605 } } peers { permanent_uuid: "9891977fbea749aeb0f0e9e80b4f9361" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 43745 } }
I20260928 20:23:19.512331 20079 leader_election.cc:288] T ae59f4c4b8344f5582f207387f028714 P 1073bb7be04e4025bd11239f3303b0e3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3638055a7dc94f378d5f16baf99f151d (127.3.18.3:33365), 9891977fbea749aeb0f0e9e80b4f9361 (127.3.18.2:43745)
W20260928 20:23:19.512701 19662 leader_election.cc:341] T ae59f4c4b8344f5582f207387f028714 P 1073bb7be04e4025bd11239f3303b0e3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9891977fbea749aeb0f0e9e80b4f9361 (127.3.18.2:43745): Not found: Tablet not found: ae59f4c4b8344f5582f207387f028714
W20260928 20:23:19.512779 19662 leader_election.cc:341] T ae59f4c4b8344f5582f207387f028714 P 1073bb7be04e4025bd11239f3303b0e3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3638055a7dc94f378d5f16baf99f151d (127.3.18.3:33365): Not found: Tablet not found: ae59f4c4b8344f5582f207387f028714
I20260928 20:23:19.512799 19662 leader_election.cc:302] T ae59f4c4b8344f5582f207387f028714 P 1073bb7be04e4025bd11239f3303b0e3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1073bb7be04e4025bd11239f3303b0e3; no voters: 3638055a7dc94f378d5f16baf99f151d, 9891977fbea749aeb0f0e9e80b4f9361
I20260928 20:23:19.512872 20079 raft_consensus.cc:2726] T ae59f4c4b8344f5582f207387f028714 P 1073bb7be04e4025bd11239f3303b0e3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 20:23:19.684216 20080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:19.683894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40865 (local address 127.3.18.1:50717)
0928 20:23:19.684062 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:19.684124 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:40865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":70,"threads_started":1}
I20260928 20:23:19.732975 20081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:19.732690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40865 (local address 127.3.18.2:38843)
0928 20:23:19.732869 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:19.732924 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:40865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":81,"threads_started":1}
I20260928 20:23:19.868559 20082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:19.868314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40865 (local address 127.3.18.3:49273)
0928 20:23:19.868465 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:19.868511 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:40865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":62,"threads_started":1}
I20260928 20:23:20.685036 20083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:20.684814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40865 (local address 127.3.18.1:39787)
0928 20:23:20.684949 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:20.684988 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:40865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":62,"threads_started":1}
I20260928 20:23:20.733816 20084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:20.733557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40865 (local address 127.3.18.2:34637)
0928 20:23:20.733719 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:20.733768 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:40865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":77,"threads_started":1}
I20260928 20:23:20.869567 20085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:20.869269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40865 (local address 127.3.18.3:42239)
0928 20:23:20.869458 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:20.869505 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:40865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":94,"threads_started":1}
I20260928 20:23:21.685883 20086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:21.685591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40865 (local address 127.3.18.1:50605)
0928 20:23:21.685794 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:21.685833 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:40865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":137,"threads_started":1}
I20260928 20:23:21.734651 20087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:21.734374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40865 (local address 127.3.18.2:42595)
0928 20:23:21.734544 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:21.734593 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:40865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":85,"threads_started":1}
I20260928 20:23:21.870368 20088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:21.870145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40865 (local address 127.3.18.3:45569)
0928 20:23:21.870274 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:21.870319 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:40865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":65,"threads_started":1}
I20260928 20:23:22.686772 20089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:22.686528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40865 (local address 127.3.18.1:58971)
0928 20:23:22.686688 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:22.686726 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:40865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":65,"threads_started":1}
I20260928 20:23:22.735612 20090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:22.735287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40865 (local address 127.3.18.2:35409)
0928 20:23:22.735460 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:22.735510 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:40865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":82,"threads_started":1}
I20260928 20:23:22.871445 20091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:22.871203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40865 (local address 127.3.18.3:39661)
0928 20:23:22.871352 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:22.871397 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:40865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":66,"threads_started":1}
I20260928 20:23:23.687515 20092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:23.687304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40865 (local address 127.3.18.1:51875)
0928 20:23:23.687434 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:23.687468 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:40865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":70,"threads_started":1}
I20260928 20:23:23.736507 20093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:23.736261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40865 (local address 127.3.18.2:43897)
0928 20:23:23.736413 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:23.736458 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:40865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":85,"threads_started":1}
I20260928 20:23:23.872373 20094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:23.872103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40865 (local address 127.3.18.3:39617)
0928 20:23:23.872275 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:23.872323 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:40865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":78,"threads_started":1}
I20260928 20:23:24.688431 20095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:24.688165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40865 (local address 127.3.18.1:38621)
0928 20:23:24.688327 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:24.688371 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:40865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":91,"threads_started":1}
I20260928 20:23:24.737429 20096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:24.737156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40865 (local address 127.3.18.2:53675)
0928 20:23:24.737330 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:24.737379 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:40865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":78,"threads_started":1}
I20260928 20:23:24.865188 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 19612
I20260928 20:23:24.870009 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 19781
I20260928 20:23:24.873229 20097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:24.873002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40865 (local address 127.3.18.3:45303)
0928 20:23:24.873138 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:24.873174 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.62:40865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":80,"threads_started":1}
I20260928 20:23:24.874745 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 19913
2026-09-28T20:23:24Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58394 ms)
[----------] 30 tests from SecurityITest (240631 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:23:24 dist-test-slave-b83v krb5kdc[20100](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:23:24 dist-test-slave-b83v krb5kdc[20100](info): set up 2 sockets
Sep 28 20:23:24 dist-test-slave-b83v krb5kdc[20100](info): commencing operation
krb5kdc: starting...
W20260928 20:23:25.947857 3144 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Sep 28 20:23:26 dist-test-slave-b83v krb5kdc[20100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:23:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:23:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.18.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab.
Entry for principal kudu/127.3.18.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab.
I20260928 20:23:26.080834 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:33765
--webserver_interface=127.3.18.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:34085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.18.62:33765
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:23:26.164553 20113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:23:26.164812 20113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:23:26.164837 20113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:23:26.166088 20113 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:23:26.166138 20113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:23:26.166152 20113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:23:26.166167 20113 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:23:26.166178 20113 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:23:26.167719 20113 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:34085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.18.62:33765
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:33765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.18.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20113
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:23:26.167980 20113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:23:26.168212 20113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:23:26.173879 20119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:23:26.174021 20113 server_base.cc:1034] running on GCE node
W20260928 20:23:26.174110 20121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:26.173923 20118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:23:26.174559 20113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:23:26.174881 20113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:23:26.176056 20113 hybrid_clock.cc:648] HybridClock initialized: now 1790627006176035 us; error 33 us; skew 500 ppm
Sep 28 20:23:26 dist-test-slave-b83v krb5kdc[20100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627006, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:23:26.179270 20113 init.cc:377] Logged in from keytab as kudu/127.3.18.62@KRBTEST.COM (short username kudu)
I20260928 20:23:26.179886 20113 webserver.cc:460] Webserver started at http://127.3.18.62:33617/ using document root <none> and password file <none>
I20260928 20:23:26.180140 20113 fs_manager.cc:359] Metadata directory not provided
I20260928 20:23:26.180187 20113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:23:26.180330 20113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:23:26.181465 20113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/minicluster-data/master-0/data/instance:
uuid: "2bd7616db4154c39a995bb741679af1b"
format_stamp: "Formatted at 2026-09-28 20:23:26 on dist-test-slave-b83v"
server_key: "b24a0db4a6a238c9592ef21aefd00c38"
server_key_iv: "70c4e7687101c4a63ad5c8d07f30d107"
server_key_version: "encryptionkey@0"
I20260928 20:23:26.181830 20113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/minicluster-data/master-0/wal/instance:
uuid: "2bd7616db4154c39a995bb741679af1b"
format_stamp: "Formatted at 2026-09-28 20:23:26 on dist-test-slave-b83v"
server_key: "b24a0db4a6a238c9592ef21aefd00c38"
server_key_iv: "70c4e7687101c4a63ad5c8d07f30d107"
server_key_version: "encryptionkey@0"
I20260928 20:23:26.183306 20113 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:23:26.184357 20128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:26.184594 20113 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 20:23:26.184723 20113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/minicluster-data/master-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/minicluster-data/master-0/wal
uuid: "2bd7616db4154c39a995bb741679af1b"
format_stamp: "Formatted at 2026-09-28 20:23:26 on dist-test-slave-b83v"
server_key: "b24a0db4a6a238c9592ef21aefd00c38"
server_key_iv: "70c4e7687101c4a63ad5c8d07f30d107"
server_key_version: "encryptionkey@0"
I20260928 20:23:26.184819 20113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:23:26.194933 20113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:23:26.197882 20113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:23:26.198100 20113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:23:26.203155 20113 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.62:33765
I20260928 20:23:26.203197 20180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.62:33765 every 8 connection(s)
I20260928 20:23:26.203877 20113 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
I20260928 20:23:26.204898 20181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:23:26.206477 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 20113
I20260928 20:23:26.206604 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/minicluster-data/master-0/wal/instance
I20260928 20:23:26.206804 3144 external_mini_cluster.cc:1428] Setting key 9860279e8c8812e37304d830c5fa2612
I20260928 20:23:26.208101 20181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bd7616db4154c39a995bb741679af1b: Bootstrap starting.
I20260928 20:23:26.209021 20181 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2bd7616db4154c39a995bb741679af1b: Neither blocks nor log segments found. Creating new log.
I20260928 20:23:26.209450 20181 log.cc:824] T 00000000000000000000000000000000 P 2bd7616db4154c39a995bb741679af1b: Log is configured to *not* fsync() on all Append() calls
I20260928 20:23:26.210317 20181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bd7616db4154c39a995bb741679af1b: No bootstrap required, opened a new log
I20260928 20:23:26.212716 20181 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2bd7616db4154c39a995bb741679af1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bd7616db4154c39a995bb741679af1b" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 33765 } }
I20260928 20:23:26.212922 20181 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2bd7616db4154c39a995bb741679af1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:23:26.212965 20181 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2bd7616db4154c39a995bb741679af1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bd7616db4154c39a995bb741679af1b, State: Initialized, Role: FOLLOWER
I20260928 20:23:26.213125 20181 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2bd7616db4154c39a995bb741679af1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bd7616db4154c39a995bb741679af1b" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 33765 } }
I20260928 20:23:26.213233 20181 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2bd7616db4154c39a995bb741679af1b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:23:26.213275 20181 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2bd7616db4154c39a995bb741679af1b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:23:26.213327 20181 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2bd7616db4154c39a995bb741679af1b [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:23:26.214179 20181 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2bd7616db4154c39a995bb741679af1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bd7616db4154c39a995bb741679af1b" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 33765 } }
I20260928 20:23:26.214315 20181 leader_election.cc:302] T 00000000000000000000000000000000 P 2bd7616db4154c39a995bb741679af1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2bd7616db4154c39a995bb741679af1b; no voters:
I20260928 20:23:26.214493 20181 leader_election.cc:288] T 00000000000000000000000000000000 P 2bd7616db4154c39a995bb741679af1b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:23:26.214581 20186 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2bd7616db4154c39a995bb741679af1b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:23:26.214793 20181 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2bd7616db4154c39a995bb741679af1b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:23:26.214835 20186 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2bd7616db4154c39a995bb741679af1b [term 1 LEADER]: Becoming Leader. State: Replica: 2bd7616db4154c39a995bb741679af1b, State: Running, Role: LEADER
I20260928 20:23:26.215073 20186 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2bd7616db4154c39a995bb741679af1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bd7616db4154c39a995bb741679af1b" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 33765 } }
Sep 28 20:23:26 dist-test-slave-b83v krb5kdc[20100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:23:26.215504 20187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bd7616db4154c39a995bb741679af1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2bd7616db4154c39a995bb741679af1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bd7616db4154c39a995bb741679af1b" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 33765 } } }
I20260928 20:23:26.215595 20187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bd7616db4154c39a995bb741679af1b [sys.catalog]: This master's current role is: LEADER
I20260928 20:23:26.215799 20188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bd7616db4154c39a995bb741679af1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2bd7616db4154c39a995bb741679af1b. Latest consensus state: current_term: 1 leader_uuid: "2bd7616db4154c39a995bb741679af1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bd7616db4154c39a995bb741679af1b" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 33765 } } }
I20260928 20:23:26.215878 20188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bd7616db4154c39a995bb741679af1b [sys.catalog]: This master's current role is: LEADER
I20260928 20:23:26.216145 20195 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:23:26.216822 20195 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:23:26.218840 20195 catalog_manager.cc:1347] Generated new cluster ID: 4ca2fd774dde48e3b6c261639745a88d
I20260928 20:23:26.218914 20195 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:23:26.220975 20184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:26.207457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52180 (local address 127.3.18.62:33765)
0928 20:23:26.207651 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:26.207664 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:26.207704 (+ 40us) server_negotiation.cc:407] Connection header received
0928 20:23:26.208428 (+ 724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:26.208439 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:26.208502 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:26.208647 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:26.209597 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:26.210913 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:26.212102 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:26.212582 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:26.212753 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:26.216060 (+ 3307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:26.216065 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:26.216065 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:23:26.216081 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:26.218696 (+ 2615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:26.219150 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:26.219154 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:26.219155 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:26.219270 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:26.219511 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:26.219512 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:26.219512 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:26.219875 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:26.220003 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:26.220572 (+ 569us) server_negotiation.cc:299] Negotiation successful
0928 20:23:26.220741 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":99,"threads_started":1}
I20260928 20:23:26.242334 20195 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:23:26.243046 20195 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:23:26.253904 20195 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2bd7616db4154c39a995bb741679af1b: Generated new TSK 0
I20260928 20:23:26.254156 20195 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.18.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab.
Entry for principal kudu/127.3.18.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab.
I20260928 20:23:26.291425 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.1:0
--local_ip_for_outbound_sockets=127.3.18.1
--webserver_interface=127.3.18.1
--webserver_port=0
--tserver_master_addrs=127.3.18.62:33765
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab
--principal=kudu/127.3.18.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:34085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:23:26.383615 20207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:23:26.383846 20207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:23:26.383875 20207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:23:26.385110 20207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:23:26.385157 20207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:23:26.385185 20207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.1
I20260928 20:23:26.386793 20207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:34085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab
--principal=kudu/127.3.18.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.18.1
--webserver_port=0
--tserver_master_addrs=127.3.18.62:33765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20207
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.1
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:23:26.387068 20207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:23:26.387384 20207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:23:26.392838 20213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:26.392849 20212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:26.392838 20215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:23:26.393051 20207 server_base.cc:1034] running on GCE node
I20260928 20:23:26.393313 20207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:23:26.393525 20207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:23:26.394716 20207 hybrid_clock.cc:648] HybridClock initialized: now 1790627006394696 us; error 34 us; skew 500 ppm
Sep 28 20:23:26 dist-test-slave-b83v krb5kdc[20100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627006, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:23:26.397850 20207 init.cc:377] Logged in from keytab as kudu/127.3.18.1@KRBTEST.COM (short username kudu)
I20260928 20:23:26.398253 20207 webserver.cc:460] Webserver started at http://127.3.18.1:34563/ using document root <none> and password file <none>
I20260928 20:23:26.398546 20207 fs_manager.cc:359] Metadata directory not provided
I20260928 20:23:26.398625 20207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:23:26.398774 20207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:23:26.399940 20207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/minicluster-data/ts-0/data/instance:
uuid: "cb711d11669c49a4ad6c12ae2e031f1b"
format_stamp: "Formatted at 2026-09-28 20:23:26 on dist-test-slave-b83v"
server_key: "d203faa148e83f0f43bba4224f4e9c64"
server_key_iv: "6f4b8b64a4f96a42aa55cefc4dce093c"
server_key_version: "encryptionkey@0"
I20260928 20:23:26.400315 20207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/minicluster-data/ts-0/wal/instance:
uuid: "cb711d11669c49a4ad6c12ae2e031f1b"
format_stamp: "Formatted at 2026-09-28 20:23:26 on dist-test-slave-b83v"
server_key: "d203faa148e83f0f43bba4224f4e9c64"
server_key_iv: "6f4b8b64a4f96a42aa55cefc4dce093c"
server_key_version: "encryptionkey@0"
I20260928 20:23:26.401749 20207 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 20:23:26.402558 20222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:26.402832 20207 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:23:26.402933 20207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/minicluster-data/ts-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/minicluster-data/ts-0/wal
uuid: "cb711d11669c49a4ad6c12ae2e031f1b"
format_stamp: "Formatted at 2026-09-28 20:23:26 on dist-test-slave-b83v"
server_key: "d203faa148e83f0f43bba4224f4e9c64"
server_key_iv: "6f4b8b64a4f96a42aa55cefc4dce093c"
server_key_version: "encryptionkey@0"
I20260928 20:23:26.403018 20207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:23:26.433677 20207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:23:26.436225 20207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:23:26.436408 20207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:23:26.436668 20207 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:23:26.437001 20207 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:23:26.437057 20207 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:26.437098 20207 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:23:26.437127 20207 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:26.442946 20207 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.1:42447
I20260928 20:23:26.443027 20335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.1:42447 every 8 connection(s)
I20260928 20:23:26.443387 20207 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
I20260928 20:23:26.447518 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 20207
I20260928 20:23:26.447634 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/minicluster-data/ts-0/wal/instance
I20260928 20:23:26.447861 3144 external_mini_cluster.cc:1428] Setting key f829d08b62c2152569918e086564b64e
Sep 28 20:23:26 dist-test-slave-b83v krb5kdc[20100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627006, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.1@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:23:26.454295 20184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:26.444350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:41931 (local address 127.3.18.62:33765)
0928 20:23:26.444417 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:26.444419 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:26.444622 (+ 203us) server_negotiation.cc:407] Connection header received
0928 20:23:26.445101 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:26.445102 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:26.445114 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:26.445186 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:26.446117 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:26.446735 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:26.448081 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:26.448386 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:26.448463 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:26.451162 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:26.451164 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:26.451164 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:23:26.451177 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:26.452709 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:26.453182 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:26.453183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:26.453184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:26.453243 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:26.453521 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:26.453522 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:26.453522 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:26.453649 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:26.453719 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:26.454053 (+ 334us) server_negotiation.cc:299] Negotiation successful
0928 20:23:26.454199 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 20:23:26.454470 20338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:26.444236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:33765 (local address 127.3.18.1:41931)
0928 20:23:26.444485 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:26.444535 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 20:23:26.445001 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:23:26.445238 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:23:26.445242 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:23:26.445823 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:23:26.446020 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:26.446025 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:26.446830 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:26.446831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:26.447991 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:26.447995 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:26.448472 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:26.448473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:26.448628 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:26.449427 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:23:26.449436 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:23:26.451036 (+ 1600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:23:26.452806 (+ 1770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:23:26.452816 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:23:26.452819 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:23:26.453082 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:23:26.453318 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:23:26.453319 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:23:26.453319 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:23:26.453444 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:23:26.453721 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:23:26.453729 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:23:26.453896 (+ 167us) client_negotiation.cc:769] Sending connection context
0928 20:23:26.454019 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 20:23:26.454206 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":101,"threads_started":1}
I20260928 20:23:26.455034 20336 heartbeater.cc:347] Connected to a master server at 127.3.18.62:33765
I20260928 20:23:26.455160 20336 heartbeater.cc:464] Registering TS with master...
I20260928 20:23:26.455392 20336 heartbeater.cc:511] Master 127.3.18.62:33765 requested a full tablet report, sending...
I20260928 20:23:26.456156 20145 ts_manager.cc:194] Registered new tserver with Master: cb711d11669c49a4ad6c12ae2e031f1b (127.3.18.1:42447)
I20260928 20:23:26.456794 20145 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.18.1@KRBTEST.COM'} at 127.3.18.1:41931
WARNING: no policy specified for kudu/127.3.18.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab.
Entry for principal kudu/127.3.18.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab.
I20260928 20:23:26.479998 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.2:0
--local_ip_for_outbound_sockets=127.3.18.2
--webserver_interface=127.3.18.2
--webserver_port=0
--tserver_master_addrs=127.3.18.62:33765
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab
--principal=kudu/127.3.18.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:34085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:23:26.565397 20341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:23:26.565624 20341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:23:26.565649 20341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:23:26.566850 20341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:23:26.566900 20341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:23:26.566927 20341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.2
I20260928 20:23:26.568486 20341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:34085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab
--principal=kudu/127.3.18.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.18.2
--webserver_port=0
--tserver_master_addrs=127.3.18.62:33765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.2
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:23:26.568742 20341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:23:26.569003 20341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:23:26.574538 20346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:26.574548 20349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:26.574551 20347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:23:26.574798 20341 server_base.cc:1034] running on GCE node
I20260928 20:23:26.575008 20341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:23:26.575284 20341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:23:26.576525 20341 hybrid_clock.cc:648] HybridClock initialized: now 1790627006576509 us; error 37 us; skew 500 ppm
Sep 28 20:23:26 dist-test-slave-b83v krb5kdc[20100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627006, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:23:26.579216 20341 init.cc:377] Logged in from keytab as kudu/127.3.18.2@KRBTEST.COM (short username kudu)
I20260928 20:23:26.579694 20341 webserver.cc:460] Webserver started at http://127.3.18.2:43623/ using document root <none> and password file <none>
I20260928 20:23:26.579972 20341 fs_manager.cc:359] Metadata directory not provided
I20260928 20:23:26.580042 20341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:23:26.580191 20341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:23:26.581332 20341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/minicluster-data/ts-1/data/instance:
uuid: "1ac9553aeb214c098b53c31e2f9d738f"
format_stamp: "Formatted at 2026-09-28 20:23:26 on dist-test-slave-b83v"
server_key: "cd91a8cdb5a300b47675656861cb889f"
server_key_iv: "91f640f974bf93c331908f49f3433e91"
server_key_version: "encryptionkey@0"
I20260928 20:23:26.581708 20341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/minicluster-data/ts-1/wal/instance:
uuid: "1ac9553aeb214c098b53c31e2f9d738f"
format_stamp: "Formatted at 2026-09-28 20:23:26 on dist-test-slave-b83v"
server_key: "cd91a8cdb5a300b47675656861cb889f"
server_key_iv: "91f640f974bf93c331908f49f3433e91"
server_key_version: "encryptionkey@0"
I20260928 20:23:26.583109 20341 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:23:26.584093 20356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:26.584301 20341 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:23:26.584406 20341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/minicluster-data/ts-1/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/minicluster-data/ts-1/wal
uuid: "1ac9553aeb214c098b53c31e2f9d738f"
format_stamp: "Formatted at 2026-09-28 20:23:26 on dist-test-slave-b83v"
server_key: "cd91a8cdb5a300b47675656861cb889f"
server_key_iv: "91f640f974bf93c331908f49f3433e91"
server_key_version: "encryptionkey@0"
I20260928 20:23:26.584481 20341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:23:26.612044 20341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:23:26.614292 20341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:23:26.614483 20341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:23:26.614769 20341 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:23:26.615146 20341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:23:26.615200 20341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:26.615242 20341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:23:26.615269 20341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:26.621438 20341 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.2:41853
I20260928 20:23:26.621491 20469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.2:41853 every 8 connection(s)
I20260928 20:23:26.621868 20341 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
I20260928 20:23:26.625775 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 20341
I20260928 20:23:26.625872 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/minicluster-data/ts-1/wal/instance
I20260928 20:23:26.626075 3144 external_mini_cluster.cc:1428] Setting key e7bb82e79f892a9e5c5f4f424be1a2b5
Sep 28 20:23:26 dist-test-slave-b83v krb5kdc[20100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627006, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.2@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:23:26.632058 20184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:26.622716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:44467 (local address 127.3.18.62:33765)
0928 20:23:26.622789 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:26.622792 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:26.623027 (+ 235us) server_negotiation.cc:407] Connection header received
0928 20:23:26.623514 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:26.623515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:26.623525 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:26.623581 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:26.624645 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:26.625157 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:26.626305 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:26.626491 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:26.626541 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:26.629084 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:26.629086 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:26.629093 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:23:26.629102 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:26.630615 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:26.631023 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:26.631024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:26.631024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:26.631083 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:26.631349 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:26.631350 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:26.631350 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:26.631470 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:26.631542 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:26.631878 (+ 336us) server_negotiation.cc:299] Negotiation successful
0928 20:23:26.631960 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 20:23:26.632279 20472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:26.622710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:33765 (local address 127.3.18.2:44467)
0928 20:23:26.622905 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:26.622944 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 20:23:26.623424 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:23:26.623638 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:23:26.623641 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:23:26.624329 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:23:26.624535 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:26.624540 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:26.625244 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:26.625245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:26.626205 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:26.626208 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:26.626553 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:26.626554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:26.626694 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:26.627326 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:23:26.627335 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:23:26.628979 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:23:26.630697 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:23:26.630705 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:23:26.630708 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:23:26.630944 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:23:26.631160 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:23:26.631161 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:23:26.631161 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:23:26.631280 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:23:26.631544 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:23:26.631552 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:23:26.631703 (+ 151us) client_negotiation.cc:769] Sending connection context
0928 20:23:26.631855 (+ 152us) client_negotiation.cc:240] Negotiation successful
0928 20:23:26.632028 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":107,"threads_started":1}
I20260928 20:23:26.632704 20470 heartbeater.cc:347] Connected to a master server at 127.3.18.62:33765
I20260928 20:23:26.632797 20470 heartbeater.cc:464] Registering TS with master...
I20260928 20:23:26.632982 20470 heartbeater.cc:511] Master 127.3.18.62:33765 requested a full tablet report, sending...
I20260928 20:23:26.633371 20145 ts_manager.cc:194] Registered new tserver with Master: 1ac9553aeb214c098b53c31e2f9d738f (127.3.18.2:41853)
I20260928 20:23:26.633766 20145 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.18.2@KRBTEST.COM'} at 127.3.18.2:44467
WARNING: no policy specified for kudu/127.3.18.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab.
Entry for principal kudu/127.3.18.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab.
I20260928 20:23:26.654435 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.3:0
--local_ip_for_outbound_sockets=127.3.18.3
--webserver_interface=127.3.18.3
--webserver_port=0
--tserver_master_addrs=127.3.18.62:33765
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab
--principal=kudu/127.3.18.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:34085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:23:26.737452 20475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:23:26.737680 20475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:23:26.737706 20475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:23:26.738821 20475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:23:26.738859 20475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:23:26.738886 20475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.3
I20260928 20:23:26.740515 20475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:34085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab
--principal=kudu/127.3.18.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.18.3
--webserver_port=0
--tserver_master_addrs=127.3.18.62:33765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.3
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:23:26.740733 20475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:23:26.740923 20475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 20:23:26.746423 20475 server_base.cc:1034] running on GCE node
W20260928 20:23:26.746392 20483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:26.746397 20481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:26.746392 20480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:23:26.746833 20475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:23:26.747095 20475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:23:26.748239 20475 hybrid_clock.cc:648] HybridClock initialized: now 1790627006748228 us; error 33 us; skew 500 ppm
Sep 28 20:23:26 dist-test-slave-b83v krb5kdc[20100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627006, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:23:26.750880 20475 init.cc:377] Logged in from keytab as kudu/127.3.18.3@KRBTEST.COM (short username kudu)
I20260928 20:23:26.751302 20475 webserver.cc:460] Webserver started at http://127.3.18.3:39283/ using document root <none> and password file <none>
I20260928 20:23:26.751505 20475 fs_manager.cc:359] Metadata directory not provided
I20260928 20:23:26.751549 20475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:23:26.751662 20475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:23:26.752869 20475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/minicluster-data/ts-2/data/instance:
uuid: "1e4acb360f044dbdb881c93083adfe6a"
format_stamp: "Formatted at 2026-09-28 20:23:26 on dist-test-slave-b83v"
server_key: "b57a4c8d1a44610fad0558d911b43973"
server_key_iv: "7a53ed7222553d3c7fe719daf1dc4626"
server_key_version: "encryptionkey@0"
I20260928 20:23:26.753185 20475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/minicluster-data/ts-2/wal/instance:
uuid: "1e4acb360f044dbdb881c93083adfe6a"
format_stamp: "Formatted at 2026-09-28 20:23:26 on dist-test-slave-b83v"
server_key: "b57a4c8d1a44610fad0558d911b43973"
server_key_iv: "7a53ed7222553d3c7fe719daf1dc4626"
server_key_version: "encryptionkey@0"
I20260928 20:23:26.754453 20475 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 20:23:26.755295 20490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:26.755456 20475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:23:26.755537 20475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/minicluster-data/ts-2/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/minicluster-data/ts-2/wal
uuid: "1e4acb360f044dbdb881c93083adfe6a"
format_stamp: "Formatted at 2026-09-28 20:23:26 on dist-test-slave-b83v"
server_key: "b57a4c8d1a44610fad0558d911b43973"
server_key_iv: "7a53ed7222553d3c7fe719daf1dc4626"
server_key_version: "encryptionkey@0"
I20260928 20:23:26.755596 20475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:23:26.773432 20475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:23:26.775959 20475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:23:26.776077 20475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:23:26.776288 20475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:23:26.776639 20475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:23:26.776678 20475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:26.776701 20475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:23:26.776736 20475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:26.782819 20475 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.3:43411
I20260928 20:23:26.782872 20603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.3:43411 every 8 connection(s)
I20260928 20:23:26.783262 20475 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
Sep 28 20:23:26 dist-test-slave-b83v krb5kdc[20100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627006, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.3@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:23:26.789866 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 20475
I20260928 20:23:26.789974 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/minicluster-data/ts-2/wal/instance
I20260928 20:23:26.790177 3144 external_mini_cluster.cc:1428] Setting key 9f5066a7306e4b25872f72f33b9e1359
I20260928 20:23:26.793416 20184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:26.784114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:52353 (local address 127.3.18.62:33765)
0928 20:23:26.784162 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:26.784164 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:26.784331 (+ 167us) server_negotiation.cc:407] Connection header received
0928 20:23:26.784847 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:26.784849 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:26.784860 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:26.784925 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:26.785895 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:26.786410 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:26.787445 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:26.787669 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:26.787743 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:26.790328 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:26.790330 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:26.790330 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:23:26.790338 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:26.792107 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:26.792518 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:26.792519 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:26.792520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:26.792626 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:26.792863 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:26.792863 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:26.792864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:26.792980 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:26.793034 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:26.793277 (+ 243us) server_negotiation.cc:299] Negotiation successful
0928 20:23:26.793347 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 20:23:26.793586 20606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:26.784056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:33765 (local address 127.3.18.3:52353)
0928 20:23:26.784218 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:26.784260 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 20:23:26.784737 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:23:26.784999 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:23:26.785003 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:23:26.785554 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:23:26.785807 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:26.785812 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:26.786486 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:26.786487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:26.787320 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:26.787322 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:26.787756 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:26.787757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:26.787937 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:26.788524 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:23:26.788532 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:23:26.790191 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:23:26.792200 (+ 2009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:23:26.792207 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:23:26.792209 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:23:26.792442 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:23:26.792694 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:23:26.792694 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:23:26.792694 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:23:26.792801 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:23:26.793036 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:23:26.793040 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:23:26.793162 (+ 122us) client_negotiation.cc:769] Sending connection context
0928 20:23:26.793246 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 20:23:26.793372 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":96,"threads_started":1}
I20260928 20:23:26.794147 20604 heartbeater.cc:347] Connected to a master server at 127.3.18.62:33765
I20260928 20:23:26.794255 20604 heartbeater.cc:464] Registering TS with master...
I20260928 20:23:26.794440 20604 heartbeater.cc:511] Master 127.3.18.62:33765 requested a full tablet report, sending...
I20260928 20:23:26.794865 20145 ts_manager.cc:194] Registered new tserver with Master: 1e4acb360f044dbdb881c93083adfe6a (127.3.18.3:43411)
I20260928 20:23:26.795288 20145 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.18.3@KRBTEST.COM'} at 127.3.18.3:52353
I20260928 20:23:26.802138 3144 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 20:23:26.810171 20184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:26.803446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52192 (local address 127.3.18.62:33765)
0928 20:23:26.803501 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:26.803503 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:26.803591 (+ 88us) server_negotiation.cc:407] Connection header received
0928 20:23:26.803631 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:26.803632 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:26.803641 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:26.803693 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:26.804496 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:26.805004 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:26.805869 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:26.806058 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:26.806148 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:26.807556 (+ 1408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:26.807557 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:26.807557 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:23:26.807564 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:26.809014 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:26.809398 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:26.809399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:26.809399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:26.809458 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:26.809628 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:26.809629 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:26.809629 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:26.809777 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:26.809852 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:26.810033 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 20:23:26.810103 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 20:23:26.812100 20145 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52192:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 20:23:26.813828 20145 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 20:23:26.821197 20618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:26.816894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54148 (local address 127.3.18.1:42447)
0928 20:23:26.817097 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:26.817103 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:26.817260 (+ 157us) server_negotiation.cc:407] Connection header received
0928 20:23:26.817364 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:26.817366 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:26.817383 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:26.817457 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:23:26.817795 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:26.818943 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:26.820235 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:26.820733 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:26.820819 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:26.820883 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:26.821057 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 20:23:26.821098 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":89,"threads_started":1}
I20260928 20:23:26.821187 20615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:26.816767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.1:42447 (local address 127.0.0.1:54148)
0928 20:23:26.817132 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:26.817173 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 20:23:26.817278 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:23:26.817552 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:23:26.817555 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:23:26.817557 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:23:26.817684 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:26.817686 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:26.819048 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:26.819049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:26.820149 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:26.820150 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:26.820832 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:26.820833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:26.820920 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:26.820934 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 20:23:26.821037 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 20:23:26.821098 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"spinlock_wait_cycles":768,"thread_start_us":105,"threads_started":1}
I20260928 20:23:26.821300 20617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:26.816767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.3:43411 (local address 127.0.0.1:60504)
0928 20:23:26.817661 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:26.817672 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:23:26.817730 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:23:26.817908 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:23:26.817908 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:23:26.817910 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:23:26.817943 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:26.817944 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:26.819135 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:26.819136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:26.820162 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:26.820165 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:26.821085 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:26.821086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:26.821147 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:26.821151 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:23:26.821194 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 20:23:26.821230 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":874,"mutex_wait_us":2,"thread_start_us":106,"threads_started":1}
I20260928 20:23:26.821358 20619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:26.816957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43622 (local address 127.3.18.2:41853)
0928 20:23:26.817139 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:26.817144 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:26.817272 (+ 128us) server_negotiation.cc:407] Connection header received
0928 20:23:26.817409 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:26.817412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:26.817429 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:26.817500 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:23:26.817787 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:26.818582 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:26.820082 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:26.820833 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:26.820904 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:26.820963 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:26.821231 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 20:23:26.821273 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":85,"threads_started":1}
I20260928 20:23:26.821664 20620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:26.817012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60504 (local address 127.3.18.3:43411)
0928 20:23:26.817502 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:26.817507 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:26.817741 (+ 234us) server_negotiation.cc:407] Connection header received
0928 20:23:26.817795 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:26.817798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:26.817814 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:26.817898 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:23:26.818003 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:26.819042 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:26.820257 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:26.820948 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:26.821007 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:26.821054 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:26.821520 (+ 466us) server_negotiation.cc:299] Negotiation successful
0928 20:23:26.821576 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":103,"threads_started":1}
I20260928 20:23:26.821573 20616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:26.816694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.2:41853 (local address 127.0.0.1:43622)
0928 20:23:26.816948 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:26.817192 (+ 244us) client_negotiation.cc:174] Beginning negotiation
0928 20:23:26.817286 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:23:26.817581 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:23:26.817582 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:23:26.817585 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:23:26.817699 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:26.817702 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:26.818691 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:26.818692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:26.819988 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:26.819991 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:26.821000 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:26.821001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:26.821067 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:26.821071 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:23:26.821133 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 20:23:26.821184 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":42,"thread_start_us":69,"threads_started":1}
I20260928 20:23:26.822229 20270 tablet_service.cc:1467] Processing CreateTablet for tablet db0e6928ddb243e1a9f9f914b7582335 (DEFAULT_TABLE table=test-table [id=dd571764eced42ee9f8f1838139579e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:23:26.822676 20270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db0e6928ddb243e1a9f9f914b7582335. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:23:26.822638 20404 tablet_service.cc:1467] Processing CreateTablet for tablet db0e6928ddb243e1a9f9f914b7582335 (DEFAULT_TABLE table=test-table [id=dd571764eced42ee9f8f1838139579e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:23:26.822952 20404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db0e6928ddb243e1a9f9f914b7582335. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:23:26.824143 20538 tablet_service.cc:1467] Processing CreateTablet for tablet db0e6928ddb243e1a9f9f914b7582335 (DEFAULT_TABLE table=test-table [id=dd571764eced42ee9f8f1838139579e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:23:26.824509 20538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db0e6928ddb243e1a9f9f914b7582335. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:23:26.826496 20622 tablet_bootstrap.cc:492] T db0e6928ddb243e1a9f9f914b7582335 P cb711d11669c49a4ad6c12ae2e031f1b: Bootstrap starting.
I20260928 20:23:26.827214 20622 tablet_bootstrap.cc:654] T db0e6928ddb243e1a9f9f914b7582335 P cb711d11669c49a4ad6c12ae2e031f1b: Neither blocks nor log segments found. Creating new log.
I20260928 20:23:26.827443 20623 tablet_bootstrap.cc:492] T db0e6928ddb243e1a9f9f914b7582335 P 1e4acb360f044dbdb881c93083adfe6a: Bootstrap starting.
I20260928 20:23:26.827900 20621 tablet_bootstrap.cc:492] T db0e6928ddb243e1a9f9f914b7582335 P 1ac9553aeb214c098b53c31e2f9d738f: Bootstrap starting.
I20260928 20:23:26.827891 20622 log.cc:824] T db0e6928ddb243e1a9f9f914b7582335 P cb711d11669c49a4ad6c12ae2e031f1b: Log is configured to *not* fsync() on all Append() calls
I20260928 20:23:26.828544 20621 tablet_bootstrap.cc:654] T db0e6928ddb243e1a9f9f914b7582335 P 1ac9553aeb214c098b53c31e2f9d738f: Neither blocks nor log segments found. Creating new log.
I20260928 20:23:26.828540 20623 tablet_bootstrap.cc:654] T db0e6928ddb243e1a9f9f914b7582335 P 1e4acb360f044dbdb881c93083adfe6a: Neither blocks nor log segments found. Creating new log.
I20260928 20:23:26.828969 20622 tablet_bootstrap.cc:492] T db0e6928ddb243e1a9f9f914b7582335 P cb711d11669c49a4ad6c12ae2e031f1b: No bootstrap required, opened a new log
I20260928 20:23:26.829051 20622 ts_tablet_manager.cc:1397] T db0e6928ddb243e1a9f9f914b7582335 P cb711d11669c49a4ad6c12ae2e031f1b: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260928 20:23:26.829103 20623 log.cc:824] T db0e6928ddb243e1a9f9f914b7582335 P 1e4acb360f044dbdb881c93083adfe6a: Log is configured to *not* fsync() on all Append() calls
I20260928 20:23:26.829166 20621 log.cc:824] T db0e6928ddb243e1a9f9f914b7582335 P 1ac9553aeb214c098b53c31e2f9d738f: Log is configured to *not* fsync() on all Append() calls
I20260928 20:23:26.829843 20621 tablet_bootstrap.cc:492] T db0e6928ddb243e1a9f9f914b7582335 P 1ac9553aeb214c098b53c31e2f9d738f: No bootstrap required, opened a new log
I20260928 20:23:26.829888 20623 tablet_bootstrap.cc:492] T db0e6928ddb243e1a9f9f914b7582335 P 1e4acb360f044dbdb881c93083adfe6a: No bootstrap required, opened a new log
I20260928 20:23:26.829937 20621 ts_tablet_manager.cc:1397] T db0e6928ddb243e1a9f9f914b7582335 P 1ac9553aeb214c098b53c31e2f9d738f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 20:23:26.829952 20623 ts_tablet_manager.cc:1397] T db0e6928ddb243e1a9f9f914b7582335 P 1e4acb360f044dbdb881c93083adfe6a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260928 20:23:26.831225 20622 raft_consensus.cc:348] T db0e6928ddb243e1a9f9f914b7582335 P cb711d11669c49a4ad6c12ae2e031f1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ac9553aeb214c098b53c31e2f9d738f" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 41853 } } peers { permanent_uuid: "cb711d11669c49a4ad6c12ae2e031f1b" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42447 } } peers { permanent_uuid: "1e4acb360f044dbdb881c93083adfe6a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 43411 } }
I20260928 20:23:26.831302 20621 raft_consensus.cc:348] T db0e6928ddb243e1a9f9f914b7582335 P 1ac9553aeb214c098b53c31e2f9d738f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ac9553aeb214c098b53c31e2f9d738f" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 41853 } } peers { permanent_uuid: "cb711d11669c49a4ad6c12ae2e031f1b" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42447 } } peers { permanent_uuid: "1e4acb360f044dbdb881c93083adfe6a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 43411 } }
I20260928 20:23:26.831444 20622 raft_consensus.cc:374] T db0e6928ddb243e1a9f9f914b7582335 P cb711d11669c49a4ad6c12ae2e031f1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:23:26.831472 20621 raft_consensus.cc:374] T db0e6928ddb243e1a9f9f914b7582335 P 1ac9553aeb214c098b53c31e2f9d738f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:23:26.831485 20622 raft_consensus.cc:729] T db0e6928ddb243e1a9f9f914b7582335 P cb711d11669c49a4ad6c12ae2e031f1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb711d11669c49a4ad6c12ae2e031f1b, State: Initialized, Role: FOLLOWER
I20260928 20:23:26.831540 20621 raft_consensus.cc:729] T db0e6928ddb243e1a9f9f914b7582335 P 1ac9553aeb214c098b53c31e2f9d738f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ac9553aeb214c098b53c31e2f9d738f, State: Initialized, Role: FOLLOWER
I20260928 20:23:26.831635 20622 consensus_queue.cc:260] T db0e6928ddb243e1a9f9f914b7582335 P cb711d11669c49a4ad6c12ae2e031f1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ac9553aeb214c098b53c31e2f9d738f" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 41853 } } peers { permanent_uuid: "cb711d11669c49a4ad6c12ae2e031f1b" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42447 } } peers { permanent_uuid: "1e4acb360f044dbdb881c93083adfe6a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 43411 } }
I20260928 20:23:26.831676 20621 consensus_queue.cc:260] T db0e6928ddb243e1a9f9f914b7582335 P 1ac9553aeb214c098b53c31e2f9d738f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ac9553aeb214c098b53c31e2f9d738f" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 41853 } } peers { permanent_uuid: "cb711d11669c49a4ad6c12ae2e031f1b" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42447 } } peers { permanent_uuid: "1e4acb360f044dbdb881c93083adfe6a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 43411 } }
I20260928 20:23:26.831995 20622 ts_tablet_manager.cc:1428] T db0e6928ddb243e1a9f9f914b7582335 P cb711d11669c49a4ad6c12ae2e031f1b: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 20:23:26.832051 20621 ts_tablet_manager.cc:1428] T db0e6928ddb243e1a9f9f914b7582335 P 1ac9553aeb214c098b53c31e2f9d738f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 20:23:26.832021 20336 heartbeater.cc:503] Master 127.3.18.62:33765 was elected leader, sending a full tablet report...
I20260928 20:23:26.831995 20623 raft_consensus.cc:348] T db0e6928ddb243e1a9f9f914b7582335 P 1e4acb360f044dbdb881c93083adfe6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ac9553aeb214c098b53c31e2f9d738f" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 41853 } } peers { permanent_uuid: "cb711d11669c49a4ad6c12ae2e031f1b" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42447 } } peers { permanent_uuid: "1e4acb360f044dbdb881c93083adfe6a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 43411 } }
I20260928 20:23:26.832157 20623 raft_consensus.cc:374] T db0e6928ddb243e1a9f9f914b7582335 P 1e4acb360f044dbdb881c93083adfe6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:23:26.832193 20623 raft_consensus.cc:729] T db0e6928ddb243e1a9f9f914b7582335 P 1e4acb360f044dbdb881c93083adfe6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e4acb360f044dbdb881c93083adfe6a, State: Initialized, Role: FOLLOWER
I20260928 20:23:26.832288 20623 consensus_queue.cc:260] T db0e6928ddb243e1a9f9f914b7582335 P 1e4acb360f044dbdb881c93083adfe6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ac9553aeb214c098b53c31e2f9d738f" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 41853 } } peers { permanent_uuid: "cb711d11669c49a4ad6c12ae2e031f1b" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42447 } } peers { permanent_uuid: "1e4acb360f044dbdb881c93083adfe6a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 43411 } }
I20260928 20:23:26.832196 20470 heartbeater.cc:503] Master 127.3.18.62:33765 was elected leader, sending a full tablet report...
I20260928 20:23:26.832543 20623 ts_tablet_manager.cc:1428] T db0e6928ddb243e1a9f9f914b7582335 P 1e4acb360f044dbdb881c93083adfe6a: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260928 20:23:26.832568 20604 heartbeater.cc:503] Master 127.3.18.62:33765 was elected leader, sending a full tablet report...
W20260928 20:23:26.872504 20471 tablet.cc:2369] T db0e6928ddb243e1a9f9f914b7582335 P 1ac9553aeb214c098b53c31e2f9d738f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 20:23:26.944218 20337 tablet.cc:2369] T db0e6928ddb243e1a9f9f914b7582335 P cb711d11669c49a4ad6c12ae2e031f1b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 20:23:27.033924 20605 tablet.cc:2369] T db0e6928ddb243e1a9f9f914b7582335 P 1e4acb360f044dbdb881c93083adfe6a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 20:23:27.063601 20628 raft_consensus.cc:482] T db0e6928ddb243e1a9f9f914b7582335 P 1ac9553aeb214c098b53c31e2f9d738f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:23:27.063807 20628 raft_consensus.cc:504] T db0e6928ddb243e1a9f9f914b7582335 P 1ac9553aeb214c098b53c31e2f9d738f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ac9553aeb214c098b53c31e2f9d738f" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 41853 } } peers { permanent_uuid: "cb711d11669c49a4ad6c12ae2e031f1b" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42447 } } peers { permanent_uuid: "1e4acb360f044dbdb881c93083adfe6a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 43411 } }
I20260928 20:23:27.064271 20628 leader_election.cc:288] T db0e6928ddb243e1a9f9f914b7582335 P 1ac9553aeb214c098b53c31e2f9d738f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb711d11669c49a4ad6c12ae2e031f1b (127.3.18.1:42447), 1e4acb360f044dbdb881c93083adfe6a (127.3.18.3:43411)
I20260928 20:23:27.067160 20618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:27.064551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:37193 (local address 127.3.18.1:42447)
0928 20:23:27.064643 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:27.064645 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:27.064654 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:23:27.064690 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:27.064691 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:27.064700 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:27.064768 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:23:27.064930 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:27.065407 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:27.066426 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:27.066792 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:27.066874 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:27.066885 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:27.067039 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 20:23:27.067077 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 20:23:27.067178 20620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:27.064549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:36265 (local address 127.3.18.3:43411)
0928 20:23:27.064605 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:27.064607 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:27.064743 (+ 136us) server_negotiation.cc:407] Connection header received
0928 20:23:27.064827 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:27.064828 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:27.064849 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:27.064916 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:23:27.065087 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:27.065537 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:27.066446 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:27.066821 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:27.066903 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:27.066913 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:27.067071 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 20:23:27.067107 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 20:23:27.067188 20472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:27.064449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.1:42447 (local address 127.3.18.2:37193)
0928 20:23:27.064528 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:27.064541 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 20:23:27.064595 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:23:27.064772 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:23:27.064773 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:23:27.064775 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:23:27.064853 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:27.064856 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:27.065501 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:27.065502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:27.066327 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:27.066330 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:27.066873 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:27.066874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:27.066962 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:27.066965 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 20:23:27.067033 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 20:23:27.067077 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 20:23:27.067209 20630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:27.064457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.3:43411 (local address 127.3.18.2:36265)
0928 20:23:27.064663 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:27.064671 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 20:23:27.064750 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:23:27.064922 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:23:27.064922 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:23:27.064924 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:23:27.064999 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:27.065000 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:27.065622 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:27.065623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:27.066350 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:27.066353 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:27.066906 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:27.066907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:27.066989 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:27.066993 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:23:27.067066 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 20:23:27.067105 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":11,"thread_start_us":108,"threads_started":1}
I20260928 20:23:27.067586 20558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db0e6928ddb243e1a9f9f914b7582335" candidate_uuid: "1ac9553aeb214c098b53c31e2f9d738f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e4acb360f044dbdb881c93083adfe6a" is_pre_election: true
I20260928 20:23:27.067800 20558 raft_consensus.cc:2454] T db0e6928ddb243e1a9f9f914b7582335 P 1e4acb360f044dbdb881c93083adfe6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ac9553aeb214c098b53c31e2f9d738f in term 0.
I20260928 20:23:27.067761 20290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db0e6928ddb243e1a9f9f914b7582335" candidate_uuid: "1ac9553aeb214c098b53c31e2f9d738f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb711d11669c49a4ad6c12ae2e031f1b" is_pre_election: true
I20260928 20:23:27.067957 20290 raft_consensus.cc:2454] T db0e6928ddb243e1a9f9f914b7582335 P cb711d11669c49a4ad6c12ae2e031f1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ac9553aeb214c098b53c31e2f9d738f in term 0.
I20260928 20:23:27.068066 20359 leader_election.cc:302] T db0e6928ddb243e1a9f9f914b7582335 P 1ac9553aeb214c098b53c31e2f9d738f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ac9553aeb214c098b53c31e2f9d738f, 1e4acb360f044dbdb881c93083adfe6a; no voters:
I20260928 20:23:27.068209 20628 raft_consensus.cc:2781] T db0e6928ddb243e1a9f9f914b7582335 P 1ac9553aeb214c098b53c31e2f9d738f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 20:23:27.068271 20628 raft_consensus.cc:482] T db0e6928ddb243e1a9f9f914b7582335 P 1ac9553aeb214c098b53c31e2f9d738f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 20:23:27.068307 20628 raft_consensus.cc:3037] T db0e6928ddb243e1a9f9f914b7582335 P 1ac9553aeb214c098b53c31e2f9d738f [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:23:27.069013 20628 raft_consensus.cc:504] T db0e6928ddb243e1a9f9f914b7582335 P 1ac9553aeb214c098b53c31e2f9d738f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ac9553aeb214c098b53c31e2f9d738f" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 41853 } } peers { permanent_uuid: "cb711d11669c49a4ad6c12ae2e031f1b" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42447 } } peers { permanent_uuid: "1e4acb360f044dbdb881c93083adfe6a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 43411 } }
I20260928 20:23:27.069166 20628 leader_election.cc:288] T db0e6928ddb243e1a9f9f914b7582335 P 1ac9553aeb214c098b53c31e2f9d738f [CANDIDATE]: Term 1 election: Requested vote from peers cb711d11669c49a4ad6c12ae2e031f1b (127.3.18.1:42447), 1e4acb360f044dbdb881c93083adfe6a (127.3.18.3:43411)
I20260928 20:23:27.069303 20290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db0e6928ddb243e1a9f9f914b7582335" candidate_uuid: "1ac9553aeb214c098b53c31e2f9d738f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb711d11669c49a4ad6c12ae2e031f1b"
I20260928 20:23:27.069373 20558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db0e6928ddb243e1a9f9f914b7582335" candidate_uuid: "1ac9553aeb214c098b53c31e2f9d738f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e4acb360f044dbdb881c93083adfe6a"
I20260928 20:23:27.069386 20290 raft_consensus.cc:3037] T db0e6928ddb243e1a9f9f914b7582335 P cb711d11669c49a4ad6c12ae2e031f1b [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:23:27.069446 20558 raft_consensus.cc:3037] T db0e6928ddb243e1a9f9f914b7582335 P 1e4acb360f044dbdb881c93083adfe6a [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:23:27.070278 20558 raft_consensus.cc:2454] T db0e6928ddb243e1a9f9f914b7582335 P 1e4acb360f044dbdb881c93083adfe6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ac9553aeb214c098b53c31e2f9d738f in term 1.
I20260928 20:23:27.070279 20290 raft_consensus.cc:2454] T db0e6928ddb243e1a9f9f914b7582335 P cb711d11669c49a4ad6c12ae2e031f1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ac9553aeb214c098b53c31e2f9d738f in term 1.
I20260928 20:23:27.070490 20359 leader_election.cc:302] T db0e6928ddb243e1a9f9f914b7582335 P 1ac9553aeb214c098b53c31e2f9d738f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ac9553aeb214c098b53c31e2f9d738f, 1e4acb360f044dbdb881c93083adfe6a; no voters:
I20260928 20:23:27.070574 20628 raft_consensus.cc:2781] T db0e6928ddb243e1a9f9f914b7582335 P 1ac9553aeb214c098b53c31e2f9d738f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:23:27.070740 20628 raft_consensus.cc:686] T db0e6928ddb243e1a9f9f914b7582335 P 1ac9553aeb214c098b53c31e2f9d738f [term 1 LEADER]: Becoming Leader. State: Replica: 1ac9553aeb214c098b53c31e2f9d738f, State: Running, Role: LEADER
I20260928 20:23:27.070859 20628 consensus_queue.cc:237] T db0e6928ddb243e1a9f9f914b7582335 P 1ac9553aeb214c098b53c31e2f9d738f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ac9553aeb214c098b53c31e2f9d738f" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 41853 } } peers { permanent_uuid: "cb711d11669c49a4ad6c12ae2e031f1b" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42447 } } peers { permanent_uuid: "1e4acb360f044dbdb881c93083adfe6a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 43411 } }
I20260928 20:23:27.072011 20143 catalog_manager.cc:5495] T db0e6928ddb243e1a9f9f914b7582335 P 1ac9553aeb214c098b53c31e2f9d738f reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ac9553aeb214c098b53c31e2f9d738f (127.3.18.2). New cstate: current_term: 1 leader_uuid: "1ac9553aeb214c098b53c31e2f9d738f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ac9553aeb214c098b53c31e2f9d738f" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 41853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb711d11669c49a4ad6c12ae2e031f1b" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e4acb360f044dbdb881c93083adfe6a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 43411 } health_report { overall_health: UNKNOWN } } }
I20260928 20:23:27.104875 20619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:27.102040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43638 (local address 127.3.18.2:41853)
0928 20:23:27.102107 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:27.102109 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:27.102118 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:23:27.102188 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:27.102188 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:27.102210 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:27.102283 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:23:27.102495 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:27.103150 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:27.104045 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:27.104244 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:27.104310 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:27.104493 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:23:27.104596 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:23:27.104665 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:27.104723 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 20:23:27.104742 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 20:23:27.107182 20290 raft_consensus.cc:1261] T db0e6928ddb243e1a9f9f914b7582335 P cb711d11669c49a4ad6c12ae2e031f1b [term 1 FOLLOWER]: Refusing update from remote peer 1ac9553aeb214c098b53c31e2f9d738f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:23:27.107216 20558 raft_consensus.cc:1261] T db0e6928ddb243e1a9f9f914b7582335 P 1e4acb360f044dbdb881c93083adfe6a [term 1 FOLLOWER]: Refusing update from remote peer 1ac9553aeb214c098b53c31e2f9d738f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:23:27.107621 20631 consensus_queue.cc:1035] T db0e6928ddb243e1a9f9f914b7582335 P 1ac9553aeb214c098b53c31e2f9d738f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e4acb360f044dbdb881c93083adfe6a" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 43411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:23:27.107872 20628 consensus_queue.cc:1035] T db0e6928ddb243e1a9f9f914b7582335 P 1ac9553aeb214c098b53c31e2f9d738f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb711d11669c49a4ad6c12ae2e031f1b" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 42447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:23:27.110565 20635 mvcc.cc:205] Tried to move back new op lower bound from 7334408221107089408 to 7334408220962848768. Current Snapshot: MvccSnapshot[applied={T|T < 7334408221107089408}]
I20260928 20:23:27.110661 20634 mvcc.cc:205] Tried to move back new op lower bound from 7334408221107089408 to 7334408220962848768. Current Snapshot: MvccSnapshot[applied={T|T < 7334408221107089408}]
I20260928 20:23:27.113159 20143 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:23:27.113278 20143 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:23:27.114133 20143 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2bd7616db4154c39a995bb741679af1b: Sending DeleteTablet for 3 replicas of tablet db0e6928ddb243e1a9f9f914b7582335
I20260928 20:23:27.114499 20270 tablet_service.cc:1514] Processing DeleteTablet for tablet db0e6928ddb243e1a9f9f914b7582335 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:23:27 UTC) from {username='kudu', principal='kudu/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:54148
I20260928 20:23:27.114655 20538 tablet_service.cc:1514] Processing DeleteTablet for tablet db0e6928ddb243e1a9f9f914b7582335 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:23:27 UTC) from {username='kudu', principal='kudu/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:60504
I20260928 20:23:27.114766 20644 tablet_replica.cc:330] T db0e6928ddb243e1a9f9f914b7582335 P cb711d11669c49a4ad6c12ae2e031f1b: stopping tablet replica
I20260928 20:23:27.114887 20644 raft_consensus.cc:2229] T db0e6928ddb243e1a9f9f914b7582335 P cb711d11669c49a4ad6c12ae2e031f1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:23:27.114923 20645 tablet_replica.cc:330] T db0e6928ddb243e1a9f9f914b7582335 P 1e4acb360f044dbdb881c93083adfe6a: stopping tablet replica
I20260928 20:23:27.114928 20404 tablet_service.cc:1514] Processing DeleteTablet for tablet db0e6928ddb243e1a9f9f914b7582335 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:23:27 UTC) from {username='kudu', principal='kudu/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:43622
I20260928 20:23:27.115020 20645 raft_consensus.cc:2229] T db0e6928ddb243e1a9f9f914b7582335 P 1e4acb360f044dbdb881c93083adfe6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:23:27.115074 20644 raft_consensus.cc:2258] T db0e6928ddb243e1a9f9f914b7582335 P cb711d11669c49a4ad6c12ae2e031f1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:23:27.115123 20645 raft_consensus.cc:2258] T db0e6928ddb243e1a9f9f914b7582335 P 1e4acb360f044dbdb881c93083adfe6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:23:27.115587 20645 ts_tablet_manager.cc:1905] T db0e6928ddb243e1a9f9f914b7582335 P 1e4acb360f044dbdb881c93083adfe6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:23:27.115607 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 20207
I20260928 20:23:27.116245 20646 tablet_replica.cc:330] T db0e6928ddb243e1a9f9f914b7582335 P 1ac9553aeb214c098b53c31e2f9d738f: stopping tablet replica
I20260928 20:23:27.116353 20646 raft_consensus.cc:2229] T db0e6928ddb243e1a9f9f914b7582335 P 1ac9553aeb214c098b53c31e2f9d738f [term 1 LEADER]: Raft consensus shutting down.
I20260928 20:23:27.116482 20646 raft_consensus.cc:2258] T db0e6928ddb243e1a9f9f914b7582335 P 1ac9553aeb214c098b53c31e2f9d738f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:23:27.117707 20645 ts_tablet_manager.cc:1918] T db0e6928ddb243e1a9f9f914b7582335 P 1e4acb360f044dbdb881c93083adfe6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:23:27.117776 20645 log.cc:1196] T db0e6928ddb243e1a9f9f914b7582335 P 1e4acb360f044dbdb881c93083adfe6a: Deleting WAL directory at /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790626764250411-3144-0/minicluster-data/ts-2/wal/wals/db0e6928ddb243e1a9f9f914b7582335
I20260928 20:23:27.118062 20645 ts_tablet_manager.cc:1939] T db0e6928ddb243e1a9f9f914b7582335 P 1e4acb360f044dbdb881c93083adfe6a: Deleting consensus metadata
I20260928 20:23:27.118659 20131 catalog_manager.cc:4841] TS 1e4acb360f044dbdb881c93083adfe6a (127.3.18.3:43411): tablet db0e6928ddb243e1a9f9f914b7582335 (table test-table [id=dd571764eced42ee9f8f1838139579e0]) successfully deleted
W20260928 20:23:27.122658 20129 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.18.1:42447 (error 108)
I20260928 20:23:27.122845 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 20341
I20260928 20:23:27.122884 20646 ts_tablet_manager.cc:1905] T db0e6928ddb243e1a9f9f914b7582335 P 1ac9553aeb214c098b53c31e2f9d738f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:23:27.124418 20616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:27.123266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.1:42447 (local address 127.0.0.1:54160)
0928 20:23:27.124337 (+ 1071us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:27.124384 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.1:42447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1048}
W20260928 20:23:27.124753 20129 catalog_manager.cc:4584] TS cb711d11669c49a4ad6c12ae2e031f1b (127.3.18.1:42447): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet db0e6928ddb243e1a9f9f914b7582335: Network error: Client connection negotiation failed: client connection to 127.3.18.1:42447: connect: Connection refused (error 111)
I20260928 20:23:27.127806 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 20475
I20260928 20:23:27.127928 20616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:27.127726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.2:41853 (local address 127.0.0.1:43654)
0928 20:23:27.127829 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:27.127860 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.2:41853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260928 20:23:27.132414 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 20113
2026-09-28T20:23:27Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2257 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T20:23:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:23:27Z Disabled control of system clock
I20260928 20:23:27.160956 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790626764250411-3144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790626764250411-3144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790626764250411-3144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:40921
--webserver_interface=127.3.18.62
--webserver_port=0
--builtin_ntp_servers=127.3.18.20:35861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.18.62:40921
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 20:23:27.244287 20651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:23:27.244557 20651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:23:27.244580 20651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:23:27.245672 20651 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:23:27.245705 20651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:23:27.245719 20651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:23:27.245731 20651 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:23:27.245743 20651 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:23:27.247011 20651 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:35861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790626764250411-3144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790626764250411-3144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.18.62:40921
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:40921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.18.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790626764250411-3144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:23:27.247253 20651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:23:27.247506 20651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:23:27.252839 20657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:23:27.253018 20651 server_base.cc:1034] running on GCE node
W20260928 20:23:27.252848 20656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:27.252902 20659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:23:27.253518 20651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:23:27.253849 20651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:23:27.255017 20651 hybrid_clock.cc:648] HybridClock initialized: now 1790627007255010 us; error 39 us; skew 500 ppm
I20260928 20:23:27.256426 20651 webserver.cc:460] Webserver started at http://127.3.18.62:37219/ using document root <none> and password file <none>
I20260928 20:23:27.256706 20651 fs_manager.cc:359] Metadata directory not provided
I20260928 20:23:27.256781 20651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:23:27.256934 20651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:23:27.258055 20651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790626764250411-3144-0/minicluster-data/master-0/data/instance:
uuid: "3f47c8b5a25d4c499d1dd4380fb21d6c"
format_stamp: "Formatted at 2026-09-28 20:23:27 on dist-test-slave-b83v"
server_key: "87f50a8a1668cafa9958e44d4d996b2d"
server_key_iv: "366b5c6194a5b52206d82b6d04038739"
server_key_version: "encryptionkey@0"
I20260928 20:23:27.258433 20651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790626764250411-3144-0/minicluster-data/master-0/wal/instance:
uuid: "3f47c8b5a25d4c499d1dd4380fb21d6c"
format_stamp: "Formatted at 2026-09-28 20:23:27 on dist-test-slave-b83v"
server_key: "87f50a8a1668cafa9958e44d4d996b2d"
server_key_iv: "366b5c6194a5b52206d82b6d04038739"
server_key_version: "encryptionkey@0"
I20260928 20:23:27.259923 20651 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:23:27.260895 20665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:27.261137 20651 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:23:27.261255 20651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790626764250411-3144-0/minicluster-data/master-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790626764250411-3144-0/minicluster-data/master-0/wal
uuid: "3f47c8b5a25d4c499d1dd4380fb21d6c"
format_stamp: "Formatted at 2026-09-28 20:23:27 on dist-test-slave-b83v"
server_key: "87f50a8a1668cafa9958e44d4d996b2d"
server_key_iv: "366b5c6194a5b52206d82b6d04038739"
server_key_version: "encryptionkey@0"
I20260928 20:23:27.261346 20651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790626764250411-3144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790626764250411-3144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790626764250411-3144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:23:27.286921 20651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:23:27.287315 20651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:23:27.287492 20651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:23:27.291543 20651 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.62:40921
I20260928 20:23:27.291633 20717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.62:40921 every 8 connection(s)
I20260928 20:23:27.292033 20651 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
I20260928 20:23:27.293036 20718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:23:27.295329 20718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f47c8b5a25d4c499d1dd4380fb21d6c: Bootstrap starting.
I20260928 20:23:27.295902 20718 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f47c8b5a25d4c499d1dd4380fb21d6c: Neither blocks nor log segments found. Creating new log.
I20260928 20:23:27.296006 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 20651
I20260928 20:23:27.296160 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790626764250411-3144-0/minicluster-data/master-0/wal/instance
I20260928 20:23:27.296367 3144 external_mini_cluster.cc:1428] Setting key addf20a03c42e0d0b372ce6767b34107
I20260928 20:23:27.296350 20718 log.cc:824] T 00000000000000000000000000000000 P 3f47c8b5a25d4c499d1dd4380fb21d6c: Log is configured to *not* fsync() on all Append() calls
I20260928 20:23:27.297183 20718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f47c8b5a25d4c499d1dd4380fb21d6c: No bootstrap required, opened a new log
I20260928 20:23:27.299691 20718 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f47c8b5a25d4c499d1dd4380fb21d6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f47c8b5a25d4c499d1dd4380fb21d6c" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 40921 } }
I20260928 20:23:27.299907 20718 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f47c8b5a25d4c499d1dd4380fb21d6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:23:27.299942 20718 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f47c8b5a25d4c499d1dd4380fb21d6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f47c8b5a25d4c499d1dd4380fb21d6c, State: Initialized, Role: FOLLOWER
I20260928 20:23:27.300093 20718 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f47c8b5a25d4c499d1dd4380fb21d6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f47c8b5a25d4c499d1dd4380fb21d6c" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 40921 } }
I20260928 20:23:27.300244 20718 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f47c8b5a25d4c499d1dd4380fb21d6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:23:27.300312 20718 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f47c8b5a25d4c499d1dd4380fb21d6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:23:27.300372 20718 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f47c8b5a25d4c499d1dd4380fb21d6c [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:23:27.301060 20718 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f47c8b5a25d4c499d1dd4380fb21d6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f47c8b5a25d4c499d1dd4380fb21d6c" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 40921 } }
I20260928 20:23:27.301198 20718 leader_election.cc:302] T 00000000000000000000000000000000 P 3f47c8b5a25d4c499d1dd4380fb21d6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f47c8b5a25d4c499d1dd4380fb21d6c; no voters:
I20260928 20:23:27.301385 20718 leader_election.cc:288] T 00000000000000000000000000000000 P 3f47c8b5a25d4c499d1dd4380fb21d6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:23:27.301457 20723 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f47c8b5a25d4c499d1dd4380fb21d6c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:23:27.301661 20723 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f47c8b5a25d4c499d1dd4380fb21d6c [term 1 LEADER]: Becoming Leader. State: Replica: 3f47c8b5a25d4c499d1dd4380fb21d6c, State: Running, Role: LEADER
I20260928 20:23:27.301829 20723 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f47c8b5a25d4c499d1dd4380fb21d6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f47c8b5a25d4c499d1dd4380fb21d6c" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 40921 } }
I20260928 20:23:27.301702 20718 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f47c8b5a25d4c499d1dd4380fb21d6c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:23:27.302235 20724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f47c8b5a25d4c499d1dd4380fb21d6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f47c8b5a25d4c499d1dd4380fb21d6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f47c8b5a25d4c499d1dd4380fb21d6c" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 40921 } } }
I20260928 20:23:27.302353 20724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f47c8b5a25d4c499d1dd4380fb21d6c [sys.catalog]: This master's current role is: LEADER
I20260928 20:23:27.302274 20725 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f47c8b5a25d4c499d1dd4380fb21d6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f47c8b5a25d4c499d1dd4380fb21d6c. Latest consensus state: current_term: 1 leader_uuid: "3f47c8b5a25d4c499d1dd4380fb21d6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f47c8b5a25d4c499d1dd4380fb21d6c" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 40921 } } }
I20260928 20:23:27.302510 20725 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f47c8b5a25d4c499d1dd4380fb21d6c [sys.catalog]: This master's current role is: LEADER
I20260928 20:23:27.302544 20722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:27.296931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56440 (local address 127.3.18.62:40921)
0928 20:23:27.297171 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:27.297179 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:27.297205 (+ 26us) server_negotiation.cc:407] Connection header received
0928 20:23:27.297690 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:27.297697 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:27.297759 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:27.297877 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:27.298125 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:27.298994 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:27.300186 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:27.300842 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:27.300962 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:27.301784 (+ 822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:27.301789 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:27.301790 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:23:27.302020 (+ 230us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:27.302063 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:23:27.302077 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:27.302141 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:27.302233 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 20:23:27.302330 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":99,"threads_started":1}
I20260928 20:23:27.302662 20727 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:23:27.303138 20727 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:23:27.308858 20727 catalog_manager.cc:1347] Generated new cluster ID: a91a69b9746349a1b215423894acbfcb
I20260928 20:23:27.308972 20727 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:23:27.324301 20727 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:23:27.325166 20727 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:23:27.334829 20727 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f47c8b5a25d4c499d1dd4380fb21d6c: Generated new TSK 0
I20260928 20:23:27.335103 20727 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 20:23:27.337764 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790626764250411-3144-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790626764250411-3144-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790626764250411-3144-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.1:0
--local_ip_for_outbound_sockets=127.3.18.1
--webserver_interface=127.3.18.1
--webserver_port=0
--tserver_master_addrs=127.3.18.62:40921
--builtin_ntp_servers=127.3.18.20:35861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 20:23:27.423147 20742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:23:27.423408 20742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:23:27.423448 20742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:23:27.424762 20742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:23:27.424844 20742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:23:27.424890 20742 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.1
I20260928 20:23:27.426362 20742 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:35861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790626764250411-3144-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790626764250411-3144-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.18.1
--webserver_port=0
--tserver_master_addrs=127.3.18.62:40921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.1
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790626764250411-3144-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:23:27.426664 20742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:23:27.426991 20742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:23:27.432317 20748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:27.432349 20747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:27.432341 20750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:23:27.432950 20742 server_base.cc:1034] running on GCE node
I20260928 20:23:27.433168 20742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:23:27.433442 20742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:23:27.434598 20742 hybrid_clock.cc:648] HybridClock initialized: now 1790627007434572 us; error 39 us; skew 500 ppm
I20260928 20:23:27.436053 20742 webserver.cc:460] Webserver started at http://127.3.18.1:45461/ using document root <none> and password file <none>
I20260928 20:23:27.436326 20742 fs_manager.cc:359] Metadata directory not provided
I20260928 20:23:27.436383 20742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:23:27.436502 20742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:23:27.437986 20742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790626764250411-3144-0/minicluster-data/ts-0/data/instance:
uuid: "9296c8b87daf42599543855dae667f68"
format_stamp: "Formatted at 2026-09-28 20:23:27 on dist-test-slave-b83v"
server_key: "9b790008b122e9107cbcaefa4a8778a5"
server_key_iv: "19824bf70492b36027169044d5a930d1"
server_key_version: "encryptionkey@0"
I20260928 20:23:27.438402 20742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790626764250411-3144-0/minicluster-data/ts-0/wal/instance:
uuid: "9296c8b87daf42599543855dae667f68"
format_stamp: "Formatted at 2026-09-28 20:23:27 on dist-test-slave-b83v"
server_key: "9b790008b122e9107cbcaefa4a8778a5"
server_key_iv: "19824bf70492b36027169044d5a930d1"
server_key_version: "encryptionkey@0"
I20260928 20:23:27.440109 20742 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260928 20:23:27.440980 20756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:27.441187 20742 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:23:27.441283 20742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790626764250411-3144-0/minicluster-data/ts-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790626764250411-3144-0/minicluster-data/ts-0/wal
uuid: "9296c8b87daf42599543855dae667f68"
format_stamp: "Formatted at 2026-09-28 20:23:27 on dist-test-slave-b83v"
server_key: "9b790008b122e9107cbcaefa4a8778a5"
server_key_iv: "19824bf70492b36027169044d5a930d1"
server_key_version: "encryptionkey@0"
I20260928 20:23:27.441358 20742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790626764250411-3144-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790626764250411-3144-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790626764250411-3144-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:23:27.457120 20742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:23:27.457402 20742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:23:27.457510 20742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:23:27.457691 20742 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:23:27.458035 20742 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:23:27.458070 20742 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:27.458101 20742 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:23:27.458148 20742 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:27.464484 20742 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.1:34231
I20260928 20:23:27.464586 20869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.1:34231 every 8 connection(s)
I20260928 20:23:27.464937 20742 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
I20260928 20:23:27.470495 20722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:27.465767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:51579 (local address 127.3.18.62:40921)
0928 20:23:27.465825 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:27.465827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:27.466078 (+ 251us) server_negotiation.cc:407] Connection header received
0928 20:23:27.466576 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:27.466577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:27.466588 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:27.466655 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:27.467331 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:27.467918 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:27.468950 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:27.469157 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:27.469226 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:27.470098 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:27.470100 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:27.470100 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:23:27.470105 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:27.470121 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:23:27.470125 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:27.470177 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:27.470352 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 20:23:27.470380 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 20:23:27.470705 20872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:27.465768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40921 (local address 127.3.18.1:51579)
0928 20:23:27.465950 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:27.465986 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 20:23:27.466470 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:23:27.466712 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:23:27.466716 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:23:27.466724 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:23:27.467248 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:27.467253 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:27.468009 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:27.468010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:27.468840 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:27.468843 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:27.469257 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:27.469258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:27.469381 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:27.469924 (+ 543us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 20:23:27.469925 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:23:27.469956 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 20:23:27.469980 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:23:27.470215 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:23:27.470223 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:23:27.470226 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 20:23:27.470309 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 20:23:27.470403 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":89,"threads_started":1}
I20260928 20:23:27.471339 20870 heartbeater.cc:347] Connected to a master server at 127.3.18.62:40921
I20260928 20:23:27.471446 20870 heartbeater.cc:464] Registering TS with master...
I20260928 20:23:27.471629 20870 heartbeater.cc:511] Master 127.3.18.62:40921 requested a full tablet report, sending...
I20260928 20:23:27.472177 20682 ts_manager.cc:194] Registered new tserver with Master: 9296c8b87daf42599543855dae667f68 (127.3.18.1:34231)
I20260928 20:23:27.472875 20682 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.18.1:51579
I20260928 20:23:27.474162 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 20742
I20260928 20:23:27.474259 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790626764250411-3144-0/minicluster-data/ts-0/wal/instance
I20260928 20:23:27.474402 3144 external_mini_cluster.cc:1428] Setting key b1532a229b08c33a569684d060ad528f
I20260928 20:23:27.475263 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790626764250411-3144-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790626764250411-3144-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790626764250411-3144-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.2:0
--local_ip_for_outbound_sockets=127.3.18.2
--webserver_interface=127.3.18.2
--webserver_port=0
--tserver_master_addrs=127.3.18.62:40921
--builtin_ntp_servers=127.3.18.20:35861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 20:23:27.559520 20873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:23:27.559856 20873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:23:27.559890 20873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:23:27.561088 20873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:23:27.561127 20873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:23:27.561153 20873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.2
I20260928 20:23:27.562476 20873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:35861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790626764250411-3144-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790626764250411-3144-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.18.2
--webserver_port=0
--tserver_master_addrs=127.3.18.62:40921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.2
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790626764250411-3144-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:23:27.562686 20873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:23:27.562942 20873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:23:27.568239 20879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:27.568231 20881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:27.568307 20878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:23:27.568343 20873 server_base.cc:1034] running on GCE node
I20260928 20:23:27.568702 20873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:23:27.568979 20873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:23:27.570132 20873 hybrid_clock.cc:648] HybridClock initialized: now 1790627007570118 us; error 31 us; skew 500 ppm
I20260928 20:23:27.571420 20873 webserver.cc:460] Webserver started at http://127.3.18.2:38177/ using document root <none> and password file <none>
I20260928 20:23:27.571712 20873 fs_manager.cc:359] Metadata directory not provided
I20260928 20:23:27.571805 20873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:23:27.571962 20873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:23:27.573041 20873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790626764250411-3144-0/minicluster-data/ts-1/data/instance:
uuid: "78cea25def104ef9858767b1bc9213e8"
format_stamp: "Formatted at 2026-09-28 20:23:27 on dist-test-slave-b83v"
server_key: "461f08c5194ee9dc2f5f442cd1a1b940"
server_key_iv: "40fce18578dddf9e616db02f65efcc45"
server_key_version: "encryptionkey@0"
I20260928 20:23:27.573401 20873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790626764250411-3144-0/minicluster-data/ts-1/wal/instance:
uuid: "78cea25def104ef9858767b1bc9213e8"
format_stamp: "Formatted at 2026-09-28 20:23:27 on dist-test-slave-b83v"
server_key: "461f08c5194ee9dc2f5f442cd1a1b940"
server_key_iv: "40fce18578dddf9e616db02f65efcc45"
server_key_version: "encryptionkey@0"
I20260928 20:23:27.574739 20873 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:23:27.575539 20887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:27.575862 20873 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:23:27.575980 20873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790626764250411-3144-0/minicluster-data/ts-1/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790626764250411-3144-0/minicluster-data/ts-1/wal
uuid: "78cea25def104ef9858767b1bc9213e8"
format_stamp: "Formatted at 2026-09-28 20:23:27 on dist-test-slave-b83v"
server_key: "461f08c5194ee9dc2f5f442cd1a1b940"
server_key_iv: "40fce18578dddf9e616db02f65efcc45"
server_key_version: "encryptionkey@0"
I20260928 20:23:27.576061 20873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790626764250411-3144-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790626764250411-3144-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790626764250411-3144-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:23:27.591583 20873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:23:27.591984 20873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:23:27.592154 20873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:23:27.592402 20873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:23:27.592770 20873 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:23:27.592830 20873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:27.592872 20873 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:23:27.592898 20873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:27.598937 20873 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.2:45945
I20260928 20:23:27.598980 21000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.2:45945 every 8 connection(s)
I20260928 20:23:27.599325 20873 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
I20260928 20:23:27.600732 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 20873
I20260928 20:23:27.600819 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790626764250411-3144-0/minicluster-data/ts-1/wal/instance
I20260928 20:23:27.601012 3144 external_mini_cluster.cc:1428] Setting key 6c3522ef3364c3f605756e06fb8b936a
I20260928 20:23:27.602219 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790626764250411-3144-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790626764250411-3144-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790626764250411-3144-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.3:0
--local_ip_for_outbound_sockets=127.3.18.3
--webserver_interface=127.3.18.3
--webserver_port=0
--tserver_master_addrs=127.3.18.62:40921
--builtin_ntp_servers=127.3.18.20:35861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 20:23:27.605578 20722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:27.600152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:50173 (local address 127.3.18.62:40921)
0928 20:23:27.600248 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:27.600250 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:27.600495 (+ 245us) server_negotiation.cc:407] Connection header received
0928 20:23:27.601099 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:27.601101 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:27.601115 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:27.601188 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:27.602015 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:27.602644 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:27.603928 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:27.604133 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:27.604202 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:27.605242 (+ 1040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:27.605244 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:27.605244 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:23:27.605250 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:27.605269 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:23:27.605274 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:27.605335 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:27.605418 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 20:23:27.605455 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 20:23:27.605675 21003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:27.600133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40921 (local address 127.3.18.2:50173)
0928 20:23:27.600334 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:27.600380 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 20:23:27.600975 (+ 595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:23:27.601264 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:23:27.601268 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:23:27.601281 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:23:27.601914 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:27.601920 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:27.602742 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:27.602743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:27.603845 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:27.603848 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:27.604224 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:27.604225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:27.604401 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:27.605058 (+ 657us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 20:23:27.605059 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:23:27.605089 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 20:23:27.605146 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:23:27.605341 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:23:27.605346 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:23:27.605348 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 20:23:27.605402 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 20:23:27.605490 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":90,"threads_started":1}
I20260928 20:23:27.606201 21001 heartbeater.cc:347] Connected to a master server at 127.3.18.62:40921
I20260928 20:23:27.606315 21001 heartbeater.cc:464] Registering TS with master...
I20260928 20:23:27.606554 21001 heartbeater.cc:511] Master 127.3.18.62:40921 requested a full tablet report, sending...
I20260928 20:23:27.606980 20682 ts_manager.cc:194] Registered new tserver with Master: 78cea25def104ef9858767b1bc9213e8 (127.3.18.2:45945)
I20260928 20:23:27.607383 20682 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.18.2:50173
W20260928 20:23:27.687752 21004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:23:27.687988 21004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:23:27.688012 21004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:23:27.689249 21004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:23:27.689296 21004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:23:27.689323 21004 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.3
I20260928 20:23:27.690734 21004 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:35861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790626764250411-3144-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790626764250411-3144-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.18.3
--webserver_port=0
--tserver_master_addrs=127.3.18.62:40921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.3
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790626764250411-3144-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:23:27.691007 21004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:23:27.691387 21004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:23:27.696537 21009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:27.696635 21012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:27.696650 21010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:23:27.696897 21004 server_base.cc:1034] running on GCE node
I20260928 20:23:27.697095 21004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:23:27.697348 21004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:23:27.698518 21004 hybrid_clock.cc:648] HybridClock initialized: now 1790627007698490 us; error 42 us; skew 500 ppm
I20260928 20:23:27.700151 21004 webserver.cc:460] Webserver started at http://127.3.18.3:43555/ using document root <none> and password file <none>
I20260928 20:23:27.700423 21004 fs_manager.cc:359] Metadata directory not provided
I20260928 20:23:27.700471 21004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:23:27.700641 21004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:23:27.701890 21004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790626764250411-3144-0/minicluster-data/ts-2/data/instance:
uuid: "bfe55dd4fd6c475e9d5ef5abc386a02b"
format_stamp: "Formatted at 2026-09-28 20:23:27 on dist-test-slave-b83v"
server_key: "4f70ee6f00a483ca0038f11e8526327b"
server_key_iv: "8646b883ef7f2f8b38c6b7410de35ffb"
server_key_version: "encryptionkey@0"
I20260928 20:23:27.702268 21004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790626764250411-3144-0/minicluster-data/ts-2/wal/instance:
uuid: "bfe55dd4fd6c475e9d5ef5abc386a02b"
format_stamp: "Formatted at 2026-09-28 20:23:27 on dist-test-slave-b83v"
server_key: "4f70ee6f00a483ca0038f11e8526327b"
server_key_iv: "8646b883ef7f2f8b38c6b7410de35ffb"
server_key_version: "encryptionkey@0"
I20260928 20:23:27.703907 21004 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 20:23:27.704946 21018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:27.705273 21004 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:23:27.705360 21004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790626764250411-3144-0/minicluster-data/ts-2/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790626764250411-3144-0/minicluster-data/ts-2/wal
uuid: "bfe55dd4fd6c475e9d5ef5abc386a02b"
format_stamp: "Formatted at 2026-09-28 20:23:27 on dist-test-slave-b83v"
server_key: "4f70ee6f00a483ca0038f11e8526327b"
server_key_iv: "8646b883ef7f2f8b38c6b7410de35ffb"
server_key_version: "encryptionkey@0"
I20260928 20:23:27.705476 21004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790626764250411-3144-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790626764250411-3144-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790626764250411-3144-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:23:27.725600 21004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:23:27.726032 21004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:23:27.726212 21004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:23:27.726485 21004 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:23:27.726871 21004 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:23:27.726930 21004 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:27.726965 21004 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:23:27.726992 21004 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:27.732841 21004 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.3:38663
I20260928 20:23:27.732896 21131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.3:38663 every 8 connection(s)
I20260928 20:23:27.733258 21004 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
I20260928 20:23:27.738324 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 21004
I20260928 20:23:27.738451 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790626764250411-3144-0/minicluster-data/ts-2/wal/instance
I20260928 20:23:27.738652 3144 external_mini_cluster.cc:1428] Setting key 655ac4452a8ea9e02a12db34af0c1851
I20260928 20:23:27.738950 20722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:27.734070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:55491 (local address 127.3.18.62:40921)
0928 20:23:27.734153 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:27.734156 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:27.734354 (+ 198us) server_negotiation.cc:407] Connection header received
0928 20:23:27.734891 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:27.734893 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:27.734907 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:27.734984 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:27.735715 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:27.736292 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:27.737326 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:27.737517 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:27.737580 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:27.738480 (+ 900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:27.738481 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:27.738481 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:23:27.738487 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:27.738526 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:23:27.738532 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:27.738581 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:27.738793 (+ 212us) server_negotiation.cc:299] Negotiation successful
0928 20:23:27.738826 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 20:23:27.739089 21134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:27.734022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:40921 (local address 127.3.18.3:55491)
0928 20:23:27.734236 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:27.734270 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 20:23:27.734768 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:23:27.735061 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:23:27.735065 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:23:27.735073 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:23:27.735608 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:27.735612 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:27.736386 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:27.736387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:27.737229 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:27.737232 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:27.737597 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:27.737598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:27.737712 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:27.738321 (+ 609us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 20:23:27.738322 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:23:27.738352 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 20:23:27.738376 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:23:27.738662 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:23:27.738669 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:23:27.738673 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:23:27.738761 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 20:23:27.738876 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":107,"threads_started":1}
I20260928 20:23:27.739626 21132 heartbeater.cc:347] Connected to a master server at 127.3.18.62:40921
I20260928 20:23:27.739727 21132 heartbeater.cc:464] Registering TS with master...
I20260928 20:23:27.739944 21132 heartbeater.cc:511] Master 127.3.18.62:40921 requested a full tablet report, sending...
I20260928 20:23:27.740305 20682 ts_manager.cc:194] Registered new tserver with Master: bfe55dd4fd6c475e9d5ef5abc386a02b (127.3.18.3:38663)
I20260928 20:23:27.740670 20682 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.18.3:55491
I20260928 20:23:27.750823 3144 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 20:23:27.755595 20722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:27.752411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56452 (local address 127.3.18.62:40921)
0928 20:23:27.752495 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:27.752497 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:27.752590 (+ 93us) server_negotiation.cc:407] Connection header received
0928 20:23:27.752665 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:27.752666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:27.752676 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:27.752733 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:27.752937 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:27.753439 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:27.754224 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:27.754435 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:27.754536 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:27.755306 (+ 770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:27.755308 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:27.755308 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:23:27.755316 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:27.755332 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:23:27.755336 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:27.755406 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:27.755468 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 20:23:27.755495 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 20:23:27.757467 20682 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56452:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 20:23:27.759021 20682 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 20:23:27.766098 21142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:27.761548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50880 (local address 127.3.18.2:45945)
0928 20:23:27.761756 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:27.761761 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:27.762012 (+ 251us) server_negotiation.cc:407] Connection header received
0928 20:23:27.762058 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:27.762060 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:27.762078 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:27.762150 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:23:27.762359 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:27.763527 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:27.764867 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:27.765668 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:27.765748 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:27.765850 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:27.765961 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 20:23:27.766005 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":127,"threads_started":1}
I20260928 20:23:27.766119 21141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:27.761423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.2:45945 (local address 127.0.0.1:50880)
0928 20:23:27.761789 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:27.761835 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 20:23:27.761920 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:23:27.762217 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:23:27.762218 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:23:27.762221 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:23:27.762287 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:27.762289 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:27.763618 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:27.763619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:27.764780 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:27.764783 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:27.765762 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:27.765764 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:27.765846 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:27.765860 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 20:23:27.765949 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 20:23:27.766009 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":109,"threads_started":1}
I20260928 20:23:27.766273 21140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:27.761473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.3:38663 (local address 127.0.0.1:51138)
0928 20:23:27.761789 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:27.761835 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 20:23:27.761920 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:23:27.762180 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:23:27.762183 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:23:27.762186 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:23:27.762263 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:27.762265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:27.763394 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:27.763395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:27.765001 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:27.765003 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:27.766002 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:27.766003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:27.766068 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:27.766072 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:23:27.766127 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 20:23:27.766177 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":58,"threads_started":1}
I20260928 20:23:27.766314 21144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:27.761869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45696 (local address 127.3.18.1:34231)
0928 20:23:27.762289 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:27.762294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:27.762499 (+ 205us) server_negotiation.cc:407] Connection header received
0928 20:23:27.762542 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:27.762544 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:27.762561 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:27.762644 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:23:27.762790 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:27.763888 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:27.765073 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:27.765754 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:27.765835 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:27.765931 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:27.766173 (+ 242us) server_negotiation.cc:299] Negotiation successful
0928 20:23:27.766224 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":115,"threads_started":1}
I20260928 20:23:27.766441 21145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:27.761746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.1:34231 (local address 127.0.0.1:45696)
0928 20:23:27.762384 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:27.762400 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 20:23:27.762445 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:23:27.762665 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:23:27.762666 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:23:27.762668 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:23:27.762706 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:27.762707 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:27.763971 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:27.763972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:27.764964 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:27.764967 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:27.765982 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:27.765983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:27.766035 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:27.766038 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 20:23:27.766093 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 20:23:27.766127 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"mutex_wait_us":289,"thread_start_us":109,"threads_started":1}
I20260928 20:23:27.766789 21143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:27.761590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51138 (local address 127.3.18.3:38663)
0928 20:23:27.761855 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:27.761860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:27.761895 (+ 35us) server_negotiation.cc:407] Connection header received
0928 20:23:27.762029 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:27.762031 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:27.762049 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:27.762140 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:23:27.762425 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:27.763289 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:27.765112 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:27.765815 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:27.765885 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:27.765966 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:27.766643 (+ 677us) server_negotiation.cc:299] Negotiation successful
0928 20:23:27.766707 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":90,"threads_started":1}
I20260928 20:23:27.767333 20804 tablet_service.cc:1467] Processing CreateTablet for tablet 9ca0f532ec4041cb95b9fe089c90d3a1 (DEFAULT_TABLE table=test-table [id=9977996281b8442d9b7c94ce87bf63b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:23:27.767303 20935 tablet_service.cc:1467] Processing CreateTablet for tablet 9ca0f532ec4041cb95b9fe089c90d3a1 (DEFAULT_TABLE table=test-table [id=9977996281b8442d9b7c94ce87bf63b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:23:27.767707 20804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ca0f532ec4041cb95b9fe089c90d3a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:23:27.767707 20935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ca0f532ec4041cb95b9fe089c90d3a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:23:27.767597 21066 tablet_service.cc:1467] Processing CreateTablet for tablet 9ca0f532ec4041cb95b9fe089c90d3a1 (DEFAULT_TABLE table=test-table [id=9977996281b8442d9b7c94ce87bf63b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:23:27.767962 21066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ca0f532ec4041cb95b9fe089c90d3a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:23:27.770552 21146 tablet_bootstrap.cc:492] T 9ca0f532ec4041cb95b9fe089c90d3a1 P 9296c8b87daf42599543855dae667f68: Bootstrap starting.
I20260928 20:23:27.771282 21147 tablet_bootstrap.cc:492] T 9ca0f532ec4041cb95b9fe089c90d3a1 P 78cea25def104ef9858767b1bc9213e8: Bootstrap starting.
I20260928 20:23:27.771391 21146 tablet_bootstrap.cc:654] T 9ca0f532ec4041cb95b9fe089c90d3a1 P 9296c8b87daf42599543855dae667f68: Neither blocks nor log segments found. Creating new log.
I20260928 20:23:27.771751 21146 log.cc:824] T 9ca0f532ec4041cb95b9fe089c90d3a1 P 9296c8b87daf42599543855dae667f68: Log is configured to *not* fsync() on all Append() calls
I20260928 20:23:27.771958 21147 tablet_bootstrap.cc:654] T 9ca0f532ec4041cb95b9fe089c90d3a1 P 78cea25def104ef9858767b1bc9213e8: Neither blocks nor log segments found. Creating new log.
I20260928 20:23:27.771939 21148 tablet_bootstrap.cc:492] T 9ca0f532ec4041cb95b9fe089c90d3a1 P bfe55dd4fd6c475e9d5ef5abc386a02b: Bootstrap starting.
I20260928 20:23:27.772392 21147 log.cc:824] T 9ca0f532ec4041cb95b9fe089c90d3a1 P 78cea25def104ef9858767b1bc9213e8: Log is configured to *not* fsync() on all Append() calls
I20260928 20:23:27.772528 21146 tablet_bootstrap.cc:492] T 9ca0f532ec4041cb95b9fe089c90d3a1 P 9296c8b87daf42599543855dae667f68: No bootstrap required, opened a new log
I20260928 20:23:27.772614 21148 tablet_bootstrap.cc:654] T 9ca0f532ec4041cb95b9fe089c90d3a1 P bfe55dd4fd6c475e9d5ef5abc386a02b: Neither blocks nor log segments found. Creating new log.
I20260928 20:23:27.772621 21146 ts_tablet_manager.cc:1397] T 9ca0f532ec4041cb95b9fe089c90d3a1 P 9296c8b87daf42599543855dae667f68: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 20:23:27.772893 21148 log.cc:824] T 9ca0f532ec4041cb95b9fe089c90d3a1 P bfe55dd4fd6c475e9d5ef5abc386a02b: Log is configured to *not* fsync() on all Append() calls
I20260928 20:23:27.773679 21147 tablet_bootstrap.cc:492] T 9ca0f532ec4041cb95b9fe089c90d3a1 P 78cea25def104ef9858767b1bc9213e8: No bootstrap required, opened a new log
I20260928 20:23:27.773674 21148 tablet_bootstrap.cc:492] T 9ca0f532ec4041cb95b9fe089c90d3a1 P bfe55dd4fd6c475e9d5ef5abc386a02b: No bootstrap required, opened a new log
I20260928 20:23:27.773746 21148 ts_tablet_manager.cc:1397] T 9ca0f532ec4041cb95b9fe089c90d3a1 P bfe55dd4fd6c475e9d5ef5abc386a02b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 20:23:27.773746 21147 ts_tablet_manager.cc:1397] T 9ca0f532ec4041cb95b9fe089c90d3a1 P 78cea25def104ef9858767b1bc9213e8: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 20:23:27.775004 21146 raft_consensus.cc:348] T 9ca0f532ec4041cb95b9fe089c90d3a1 P 9296c8b87daf42599543855dae667f68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78cea25def104ef9858767b1bc9213e8" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45945 } } peers { permanent_uuid: "9296c8b87daf42599543855dae667f68" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34231 } } peers { permanent_uuid: "bfe55dd4fd6c475e9d5ef5abc386a02b" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38663 } }
I20260928 20:23:27.775141 21146 raft_consensus.cc:374] T 9ca0f532ec4041cb95b9fe089c90d3a1 P 9296c8b87daf42599543855dae667f68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:23:27.775163 21146 raft_consensus.cc:729] T 9ca0f532ec4041cb95b9fe089c90d3a1 P 9296c8b87daf42599543855dae667f68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9296c8b87daf42599543855dae667f68, State: Initialized, Role: FOLLOWER
I20260928 20:23:27.775288 21146 consensus_queue.cc:260] T 9ca0f532ec4041cb95b9fe089c90d3a1 P 9296c8b87daf42599543855dae667f68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78cea25def104ef9858767b1bc9213e8" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45945 } } peers { permanent_uuid: "9296c8b87daf42599543855dae667f68" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34231 } } peers { permanent_uuid: "bfe55dd4fd6c475e9d5ef5abc386a02b" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38663 } }
I20260928 20:23:27.775573 21146 ts_tablet_manager.cc:1428] T 9ca0f532ec4041cb95b9fe089c90d3a1 P 9296c8b87daf42599543855dae667f68: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 20:23:27.775619 20870 heartbeater.cc:503] Master 127.3.18.62:40921 was elected leader, sending a full tablet report...
I20260928 20:23:27.776764 21147 raft_consensus.cc:348] T 9ca0f532ec4041cb95b9fe089c90d3a1 P 78cea25def104ef9858767b1bc9213e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78cea25def104ef9858767b1bc9213e8" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45945 } } peers { permanent_uuid: "9296c8b87daf42599543855dae667f68" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34231 } } peers { permanent_uuid: "bfe55dd4fd6c475e9d5ef5abc386a02b" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38663 } }
I20260928 20:23:27.776743 21148 raft_consensus.cc:348] T 9ca0f532ec4041cb95b9fe089c90d3a1 P bfe55dd4fd6c475e9d5ef5abc386a02b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78cea25def104ef9858767b1bc9213e8" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45945 } } peers { permanent_uuid: "9296c8b87daf42599543855dae667f68" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34231 } } peers { permanent_uuid: "bfe55dd4fd6c475e9d5ef5abc386a02b" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38663 } }
I20260928 20:23:27.776953 21147 raft_consensus.cc:374] T 9ca0f532ec4041cb95b9fe089c90d3a1 P 78cea25def104ef9858767b1bc9213e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:23:27.776953 21148 raft_consensus.cc:374] T 9ca0f532ec4041cb95b9fe089c90d3a1 P bfe55dd4fd6c475e9d5ef5abc386a02b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:23:27.776993 21148 raft_consensus.cc:729] T 9ca0f532ec4041cb95b9fe089c90d3a1 P bfe55dd4fd6c475e9d5ef5abc386a02b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfe55dd4fd6c475e9d5ef5abc386a02b, State: Initialized, Role: FOLLOWER
I20260928 20:23:27.776993 21147 raft_consensus.cc:729] T 9ca0f532ec4041cb95b9fe089c90d3a1 P 78cea25def104ef9858767b1bc9213e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78cea25def104ef9858767b1bc9213e8, State: Initialized, Role: FOLLOWER
I20260928 20:23:27.777128 21147 consensus_queue.cc:260] T 9ca0f532ec4041cb95b9fe089c90d3a1 P 78cea25def104ef9858767b1bc9213e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78cea25def104ef9858767b1bc9213e8" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45945 } } peers { permanent_uuid: "9296c8b87daf42599543855dae667f68" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34231 } } peers { permanent_uuid: "bfe55dd4fd6c475e9d5ef5abc386a02b" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38663 } }
I20260928 20:23:27.777252 21148 consensus_queue.cc:260] T 9ca0f532ec4041cb95b9fe089c90d3a1 P bfe55dd4fd6c475e9d5ef5abc386a02b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78cea25def104ef9858767b1bc9213e8" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45945 } } peers { permanent_uuid: "9296c8b87daf42599543855dae667f68" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34231 } } peers { permanent_uuid: "bfe55dd4fd6c475e9d5ef5abc386a02b" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38663 } }
I20260928 20:23:27.777436 21147 ts_tablet_manager.cc:1428] T 9ca0f532ec4041cb95b9fe089c90d3a1 P 78cea25def104ef9858767b1bc9213e8: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260928 20:23:27.777498 21001 heartbeater.cc:503] Master 127.3.18.62:40921 was elected leader, sending a full tablet report...
I20260928 20:23:27.777643 21148 ts_tablet_manager.cc:1428] T 9ca0f532ec4041cb95b9fe089c90d3a1 P bfe55dd4fd6c475e9d5ef5abc386a02b: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260928 20:23:27.777765 21132 heartbeater.cc:503] Master 127.3.18.62:40921 was elected leader, sending a full tablet report...
I20260928 20:23:27.792438 21152 raft_consensus.cc:482] T 9ca0f532ec4041cb95b9fe089c90d3a1 P 9296c8b87daf42599543855dae667f68 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:23:27.792620 21152 raft_consensus.cc:504] T 9ca0f532ec4041cb95b9fe089c90d3a1 P 9296c8b87daf42599543855dae667f68 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78cea25def104ef9858767b1bc9213e8" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45945 } } peers { permanent_uuid: "9296c8b87daf42599543855dae667f68" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34231 } } peers { permanent_uuid: "bfe55dd4fd6c475e9d5ef5abc386a02b" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38663 } }
I20260928 20:23:27.793067 21152 leader_election.cc:288] T 9ca0f532ec4041cb95b9fe089c90d3a1 P 9296c8b87daf42599543855dae667f68 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78cea25def104ef9858767b1bc9213e8 (127.3.18.2:45945), bfe55dd4fd6c475e9d5ef5abc386a02b (127.3.18.3:38663)
I20260928 20:23:27.797283 21143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:27.793414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:34509 (local address 127.3.18.3:38663)
0928 20:23:27.793468 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:27.793471 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:27.793482 (+ 11us) server_negotiation.cc:407] Connection header received
0928 20:23:27.793517 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:27.793518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:27.793530 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:27.793678 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:23:27.793831 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:27.794631 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:27.796290 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:27.796871 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:27.796962 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:27.796971 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:27.797158 (+ 187us) server_negotiation.cc:299] Negotiation successful
0928 20:23:27.797211 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 20:23:27.797276 20872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:27.793298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.3:38663 (local address 127.3.18.1:34509)
0928 20:23:27.793354 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:27.793366 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:23:27.793415 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:23:27.793678 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:23:27.793678 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:23:27.793679 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:23:27.793739 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:27.793741 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:27.794778 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:27.794780 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:27.796162 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:27.796167 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:27.796986 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:27.796987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:27.797075 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:27.797079 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:23:27.797126 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 20:23:27.797172 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":5}
I20260928 20:23:27.797362 21142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:27.793495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:60863 (local address 127.3.18.2:45945)
0928 20:23:27.793540 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:27.793542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:27.793579 (+ 37us) server_negotiation.cc:407] Connection header received
0928 20:23:27.793615 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:27.793616 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:27.793632 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:27.793691 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:23:27.793943 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:27.794665 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:27.796290 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:27.796872 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:27.796955 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:27.796966 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:27.797242 (+ 276us) server_negotiation.cc:299] Negotiation successful
0928 20:23:27.797290 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 20:23:27.797415 21155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:27.793329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.2:45945 (local address 127.3.18.1:60863)
0928 20:23:27.793518 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:27.793527 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 20:23:27.793576 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:23:27.793805 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:23:27.793806 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:23:27.793807 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:23:27.793845 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:27.793845 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:27.794778 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:27.794780 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:27.796202 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:27.796204 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:27.797031 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:27.797032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:27.797153 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:27.797157 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:23:27.797217 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 20:23:27.797255 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":104,"thread_start_us":131,"threads_started":1}
I20260928 20:23:27.797700 21086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ca0f532ec4041cb95b9fe089c90d3a1" candidate_uuid: "9296c8b87daf42599543855dae667f68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfe55dd4fd6c475e9d5ef5abc386a02b" is_pre_election: true
I20260928 20:23:27.797899 21086 raft_consensus.cc:2454] T 9ca0f532ec4041cb95b9fe089c90d3a1 P bfe55dd4fd6c475e9d5ef5abc386a02b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9296c8b87daf42599543855dae667f68 in term 0.
I20260928 20:23:27.798214 20759 leader_election.cc:302] T 9ca0f532ec4041cb95b9fe089c90d3a1 P 9296c8b87daf42599543855dae667f68 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9296c8b87daf42599543855dae667f68, bfe55dd4fd6c475e9d5ef5abc386a02b; no voters:
I20260928 20:23:27.798225 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ca0f532ec4041cb95b9fe089c90d3a1" candidate_uuid: "9296c8b87daf42599543855dae667f68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78cea25def104ef9858767b1bc9213e8" is_pre_election: true
I20260928 20:23:27.798357 20955 raft_consensus.cc:2454] T 9ca0f532ec4041cb95b9fe089c90d3a1 P 78cea25def104ef9858767b1bc9213e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9296c8b87daf42599543855dae667f68 in term 0.
I20260928 20:23:27.798386 21152 raft_consensus.cc:2781] T 9ca0f532ec4041cb95b9fe089c90d3a1 P 9296c8b87daf42599543855dae667f68 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 20:23:27.798482 21152 raft_consensus.cc:482] T 9ca0f532ec4041cb95b9fe089c90d3a1 P 9296c8b87daf42599543855dae667f68 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 20:23:27.798513 21152 raft_consensus.cc:3037] T 9ca0f532ec4041cb95b9fe089c90d3a1 P 9296c8b87daf42599543855dae667f68 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:23:27.799497 21152 raft_consensus.cc:504] T 9ca0f532ec4041cb95b9fe089c90d3a1 P 9296c8b87daf42599543855dae667f68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78cea25def104ef9858767b1bc9213e8" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45945 } } peers { permanent_uuid: "9296c8b87daf42599543855dae667f68" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34231 } } peers { permanent_uuid: "bfe55dd4fd6c475e9d5ef5abc386a02b" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38663 } }
I20260928 20:23:27.799652 21152 leader_election.cc:288] T 9ca0f532ec4041cb95b9fe089c90d3a1 P 9296c8b87daf42599543855dae667f68 [CANDIDATE]: Term 1 election: Requested vote from peers 78cea25def104ef9858767b1bc9213e8 (127.3.18.2:45945), bfe55dd4fd6c475e9d5ef5abc386a02b (127.3.18.3:38663)
I20260928 20:23:27.799795 20955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ca0f532ec4041cb95b9fe089c90d3a1" candidate_uuid: "9296c8b87daf42599543855dae667f68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78cea25def104ef9858767b1bc9213e8"
I20260928 20:23:27.799849 21086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ca0f532ec4041cb95b9fe089c90d3a1" candidate_uuid: "9296c8b87daf42599543855dae667f68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfe55dd4fd6c475e9d5ef5abc386a02b"
I20260928 20:23:27.799882 20955 raft_consensus.cc:3037] T 9ca0f532ec4041cb95b9fe089c90d3a1 P 78cea25def104ef9858767b1bc9213e8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:23:27.799940 21086 raft_consensus.cc:3037] T 9ca0f532ec4041cb95b9fe089c90d3a1 P bfe55dd4fd6c475e9d5ef5abc386a02b [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:23:27.800688 21086 raft_consensus.cc:2454] T 9ca0f532ec4041cb95b9fe089c90d3a1 P bfe55dd4fd6c475e9d5ef5abc386a02b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9296c8b87daf42599543855dae667f68 in term 1.
I20260928 20:23:27.800699 20955 raft_consensus.cc:2454] T 9ca0f532ec4041cb95b9fe089c90d3a1 P 78cea25def104ef9858767b1bc9213e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9296c8b87daf42599543855dae667f68 in term 1.
I20260928 20:23:27.800895 20758 leader_election.cc:302] T 9ca0f532ec4041cb95b9fe089c90d3a1 P 9296c8b87daf42599543855dae667f68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78cea25def104ef9858767b1bc9213e8, 9296c8b87daf42599543855dae667f68; no voters:
I20260928 20:23:27.800990 21152 raft_consensus.cc:2781] T 9ca0f532ec4041cb95b9fe089c90d3a1 P 9296c8b87daf42599543855dae667f68 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:23:27.801143 21152 raft_consensus.cc:686] T 9ca0f532ec4041cb95b9fe089c90d3a1 P 9296c8b87daf42599543855dae667f68 [term 1 LEADER]: Becoming Leader. State: Replica: 9296c8b87daf42599543855dae667f68, State: Running, Role: LEADER
I20260928 20:23:27.801244 21152 consensus_queue.cc:237] T 9ca0f532ec4041cb95b9fe089c90d3a1 P 9296c8b87daf42599543855dae667f68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78cea25def104ef9858767b1bc9213e8" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45945 } } peers { permanent_uuid: "9296c8b87daf42599543855dae667f68" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34231 } } peers { permanent_uuid: "bfe55dd4fd6c475e9d5ef5abc386a02b" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38663 } }
I20260928 20:23:27.802088 20681 catalog_manager.cc:5495] T 9ca0f532ec4041cb95b9fe089c90d3a1 P 9296c8b87daf42599543855dae667f68 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9296c8b87daf42599543855dae667f68 (127.3.18.1). New cstate: current_term: 1 leader_uuid: "9296c8b87daf42599543855dae667f68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78cea25def104ef9858767b1bc9213e8" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9296c8b87daf42599543855dae667f68" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 34231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bfe55dd4fd6c475e9d5ef5abc386a02b" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38663 } health_report { overall_health: UNKNOWN } } }
I20260928 20:23:27.814191 21144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:27.811444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45706 (local address 127.3.18.1:34231)
0928 20:23:27.811508 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:27.811510 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:27.811518 (+ 8us) server_negotiation.cc:407] Connection header received
0928 20:23:27.811608 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:27.811609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:27.811629 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:27.811694 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:23:27.811865 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:27.812342 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:27.813206 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:27.813406 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:27.813472 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:27.813647 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:23:27.813752 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:23:27.813829 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:27.814062 (+ 233us) server_negotiation.cc:299] Negotiation successful
0928 20:23:27.814094 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 20:23:27.816478 20955 raft_consensus.cc:1261] T 9ca0f532ec4041cb95b9fe089c90d3a1 P 78cea25def104ef9858767b1bc9213e8 [term 1 FOLLOWER]: Refusing update from remote peer 9296c8b87daf42599543855dae667f68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:23:27.816772 21086 raft_consensus.cc:1261] T 9ca0f532ec4041cb95b9fe089c90d3a1 P bfe55dd4fd6c475e9d5ef5abc386a02b [term 1 FOLLOWER]: Refusing update from remote peer 9296c8b87daf42599543855dae667f68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:23:27.816895 21156 consensus_queue.cc:1035] T 9ca0f532ec4041cb95b9fe089c90d3a1 P 9296c8b87daf42599543855dae667f68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78cea25def104ef9858767b1bc9213e8" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 45945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:23:27.817070 21152 consensus_queue.cc:1035] T 9ca0f532ec4041cb95b9fe089c90d3a1 P 9296c8b87daf42599543855dae667f68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfe55dd4fd6c475e9d5ef5abc386a02b" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:23:27.819825 21160 mvcc.cc:205] Tried to move back new op lower bound from 7334408224012914688 to 7334408223954542592. Current Snapshot: MvccSnapshot[applied={T|T < 7334408224012914688}]
I20260928 20:23:27.820137 21161 mvcc.cc:205] Tried to move back new op lower bound from 7334408224012914688 to 7334408223954542592. Current Snapshot: MvccSnapshot[applied={T|T < 7334408224012914688 or (T in {7334408224012914688})}]
I20260928 20:23:27.822757 20682 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56452:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:23:27.822891 20682 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56452:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:23:27.823792 20682 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3f47c8b5a25d4c499d1dd4380fb21d6c: Sending DeleteTablet for 3 replicas of tablet 9ca0f532ec4041cb95b9fe089c90d3a1
I20260928 20:23:27.824157 20804 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ca0f532ec4041cb95b9fe089c90d3a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:23:27 UTC) from {username='slave'} at 127.0.0.1:45696
I20260928 20:23:27.824163 21066 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ca0f532ec4041cb95b9fe089c90d3a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:23:27 UTC) from {username='slave'} at 127.0.0.1:51138
I20260928 20:23:27.824231 20935 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ca0f532ec4041cb95b9fe089c90d3a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:23:27 UTC) from {username='slave'} at 127.0.0.1:50880
I20260928 20:23:27.824429 21170 tablet_replica.cc:330] T 9ca0f532ec4041cb95b9fe089c90d3a1 P 78cea25def104ef9858767b1bc9213e8: stopping tablet replica
I20260928 20:23:27.824532 21168 tablet_replica.cc:330] T 9ca0f532ec4041cb95b9fe089c90d3a1 P bfe55dd4fd6c475e9d5ef5abc386a02b: stopping tablet replica
I20260928 20:23:27.824635 21170 raft_consensus.cc:2229] T 9ca0f532ec4041cb95b9fe089c90d3a1 P 78cea25def104ef9858767b1bc9213e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:23:27.824671 21169 tablet_replica.cc:330] T 9ca0f532ec4041cb95b9fe089c90d3a1 P 9296c8b87daf42599543855dae667f68: stopping tablet replica
I20260928 20:23:27.824741 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 20742
I20260928 20:23:27.824786 21170 raft_consensus.cc:2258] T 9ca0f532ec4041cb95b9fe089c90d3a1 P 78cea25def104ef9858767b1bc9213e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:23:27.824891 21168 raft_consensus.cc:2229] T 9ca0f532ec4041cb95b9fe089c90d3a1 P bfe55dd4fd6c475e9d5ef5abc386a02b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:23:27.826644 21168 raft_consensus.cc:2258] T 9ca0f532ec4041cb95b9fe089c90d3a1 P bfe55dd4fd6c475e9d5ef5abc386a02b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:23:27.826772 21170 ts_tablet_manager.cc:1905] T 9ca0f532ec4041cb95b9fe089c90d3a1 P 78cea25def104ef9858767b1bc9213e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:23:27.827070 21168 ts_tablet_manager.cc:1905] T 9ca0f532ec4041cb95b9fe089c90d3a1 P bfe55dd4fd6c475e9d5ef5abc386a02b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:23:27.828850 21168 ts_tablet_manager.cc:1918] T 9ca0f532ec4041cb95b9fe089c90d3a1 P bfe55dd4fd6c475e9d5ef5abc386a02b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:23:27.828868 21170 ts_tablet_manager.cc:1918] T 9ca0f532ec4041cb95b9fe089c90d3a1 P 78cea25def104ef9858767b1bc9213e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:23:27.828955 21170 log.cc:1196] T 9ca0f532ec4041cb95b9fe089c90d3a1 P 78cea25def104ef9858767b1bc9213e8: Deleting WAL directory at /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790626764250411-3144-0/minicluster-data/ts-1/wal/wals/9ca0f532ec4041cb95b9fe089c90d3a1
I20260928 20:23:27.828958 21168 log.cc:1196] T 9ca0f532ec4041cb95b9fe089c90d3a1 P bfe55dd4fd6c475e9d5ef5abc386a02b: Deleting WAL directory at /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790626764250411-3144-0/minicluster-data/ts-2/wal/wals/9ca0f532ec4041cb95b9fe089c90d3a1
I20260928 20:23:27.829254 21168 ts_tablet_manager.cc:1939] T 9ca0f532ec4041cb95b9fe089c90d3a1 P bfe55dd4fd6c475e9d5ef5abc386a02b: Deleting consensus metadata
I20260928 20:23:27.829265 21170 ts_tablet_manager.cc:1939] T 9ca0f532ec4041cb95b9fe089c90d3a1 P 78cea25def104ef9858767b1bc9213e8: Deleting consensus metadata
I20260928 20:23:27.829782 20668 catalog_manager.cc:4841] TS bfe55dd4fd6c475e9d5ef5abc386a02b (127.3.18.3:38663): tablet 9ca0f532ec4041cb95b9fe089c90d3a1 (table test-table [id=9977996281b8442d9b7c94ce87bf63b5]) successfully deleted
I20260928 20:23:27.829805 20667 catalog_manager.cc:4841] TS 78cea25def104ef9858767b1bc9213e8 (127.3.18.2:45945): tablet 9ca0f532ec4041cb95b9fe089c90d3a1 (table test-table [id=9977996281b8442d9b7c94ce87bf63b5]) successfully deleted
W20260928 20:23:27.830695 20667 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.18.1:34231 (error 108)
I20260928 20:23:27.830909 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 20873
I20260928 20:23:27.832019 21145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:27.831868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.1:34231 (local address 127.0.0.1:45716)
0928 20:23:27.831933 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:27.831986 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.1:34231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 20:23:27.832175 20667 catalog_manager.cc:4584] TS 9296c8b87daf42599543855dae667f68 (127.3.18.1:34231): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9ca0f532ec4041cb95b9fe089c90d3a1: Network error: Client connection negotiation failed: client connection to 127.3.18.1:34231: connect: Connection refused (error 111)
I20260928 20:23:27.836148 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 21004
I20260928 20:23:27.840853 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 20651
2026-09-28T20:23:27Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (707 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:23:27 dist-test-slave-b83v krb5kdc[21174](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:23:27 dist-test-slave-b83v krb5kdc[21174](info): set up 2 sockets
Sep 28 20:23:27 dist-test-slave-b83v krb5kdc[21174](info): commencing operation
krb5kdc: starting...
W20260928 20:23:28.879948 3144 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Sep 28 20:23:28 dist-test-slave-b83v krb5kdc[21174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:23:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:23:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.18.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab.
Entry for principal kudu/127.3.18.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab.
I20260928 20:23:29.008060 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:42701
--webserver_interface=127.3.18.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:38827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.18.62:42701
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:23:29.091655 21187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:23:29.091926 21187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:23:29.091952 21187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:23:29.093284 21187 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:23:29.093340 21187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:23:29.093357 21187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:23:29.093372 21187 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:23:29.093386 21187 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:23:29.095026 21187 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:38827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.18.62:42701
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:42701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.18.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:23:29.095309 21187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:23:29.095582 21187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:23:29.101037 21193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:29.101037 21195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:29.101161 21192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:23:29.101281 21187 server_base.cc:1034] running on GCE node
I20260928 20:23:29.101552 21187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:23:29.101938 21187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:23:29.103111 21187 hybrid_clock.cc:648] HybridClock initialized: now 1790627009103098 us; error 34 us; skew 500 ppm
Sep 28 20:23:29 dist-test-slave-b83v krb5kdc[21174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627009, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:23:29.105805 21187 init.cc:377] Logged in from keytab as kudu/127.3.18.62@KRBTEST.COM (short username kudu)
I20260928 20:23:29.106348 21187 webserver.cc:460] Webserver started at http://127.3.18.62:41993/ using document root <none> and password file <none>
I20260928 20:23:29.106616 21187 fs_manager.cc:359] Metadata directory not provided
I20260928 20:23:29.106685 21187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:23:29.106845 21187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:23:29.108011 21187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/minicluster-data/master-0/data/instance:
uuid: "4ea0363ba4334acc838268facd720683"
format_stamp: "Formatted at 2026-09-28 20:23:29 on dist-test-slave-b83v"
server_key: "31242f3e214c4523a83741894f240286"
server_key_iv: "8dddab08887a9bfcb4e3c601f5b0f657"
server_key_version: "encryptionkey@0"
I20260928 20:23:29.108408 21187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/minicluster-data/master-0/wal/instance:
uuid: "4ea0363ba4334acc838268facd720683"
format_stamp: "Formatted at 2026-09-28 20:23:29 on dist-test-slave-b83v"
server_key: "31242f3e214c4523a83741894f240286"
server_key_iv: "8dddab08887a9bfcb4e3c601f5b0f657"
server_key_version: "encryptionkey@0"
I20260928 20:23:29.109784 21187 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:23:29.110550 21202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:29.110792 21187 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:23:29.110883 21187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/minicluster-data/master-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/minicluster-data/master-0/wal
uuid: "4ea0363ba4334acc838268facd720683"
format_stamp: "Formatted at 2026-09-28 20:23:29 on dist-test-slave-b83v"
server_key: "31242f3e214c4523a83741894f240286"
server_key_iv: "8dddab08887a9bfcb4e3c601f5b0f657"
server_key_version: "encryptionkey@0"
I20260928 20:23:29.110935 21187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:23:29.120332 21187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:23:29.122707 21187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:23:29.122833 21187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:23:29.126708 21187 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.62:42701
I20260928 20:23:29.126767 21254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.62:42701 every 8 connection(s)
I20260928 20:23:29.127156 21187 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
I20260928 20:23:29.128162 21255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:23:29.130455 21255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ea0363ba4334acc838268facd720683: Bootstrap starting.
I20260928 20:23:29.131063 21255 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ea0363ba4334acc838268facd720683: Neither blocks nor log segments found. Creating new log.
I20260928 20:23:29.131369 21255 log.cc:824] T 00000000000000000000000000000000 P 4ea0363ba4334acc838268facd720683: Log is configured to *not* fsync() on all Append() calls
I20260928 20:23:29.132210 21255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ea0363ba4334acc838268facd720683: No bootstrap required, opened a new log
I20260928 20:23:29.133239 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 21187
I20260928 20:23:29.133409 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/minicluster-data/master-0/wal/instance
I20260928 20:23:29.133634 3144 external_mini_cluster.cc:1428] Setting key 1b0e05140b666f09821d6ba3650e28ac
I20260928 20:23:29.134027 21255 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ea0363ba4334acc838268facd720683 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ea0363ba4334acc838268facd720683" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 42701 } }
I20260928 20:23:29.134228 21255 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ea0363ba4334acc838268facd720683 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:23:29.134315 21255 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ea0363ba4334acc838268facd720683 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ea0363ba4334acc838268facd720683, State: Initialized, Role: FOLLOWER
I20260928 20:23:29.134521 21255 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ea0363ba4334acc838268facd720683 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ea0363ba4334acc838268facd720683" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 42701 } }
I20260928 20:23:29.134617 21255 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ea0363ba4334acc838268facd720683 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:23:29.134663 21255 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ea0363ba4334acc838268facd720683 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:23:29.134734 21255 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ea0363ba4334acc838268facd720683 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:23:29.135659 21255 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ea0363ba4334acc838268facd720683 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ea0363ba4334acc838268facd720683" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 42701 } }
I20260928 20:23:29.135805 21255 leader_election.cc:302] T 00000000000000000000000000000000 P 4ea0363ba4334acc838268facd720683 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ea0363ba4334acc838268facd720683; no voters:
I20260928 20:23:29.135998 21255 leader_election.cc:288] T 00000000000000000000000000000000 P 4ea0363ba4334acc838268facd720683 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:23:29.136049 21260 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ea0363ba4334acc838268facd720683 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:23:29.136189 21260 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ea0363ba4334acc838268facd720683 [term 1 LEADER]: Becoming Leader. State: Replica: 4ea0363ba4334acc838268facd720683, State: Running, Role: LEADER
I20260928 20:23:29.136272 21255 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ea0363ba4334acc838268facd720683 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:23:29.136277 21260 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ea0363ba4334acc838268facd720683 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ea0363ba4334acc838268facd720683" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 42701 } }
I20260928 20:23:29.136823 21262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ea0363ba4334acc838268facd720683 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ea0363ba4334acc838268facd720683. Latest consensus state: current_term: 1 leader_uuid: "4ea0363ba4334acc838268facd720683" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ea0363ba4334acc838268facd720683" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 42701 } } }
I20260928 20:23:29.136941 21262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ea0363ba4334acc838268facd720683 [sys.catalog]: This master's current role is: LEADER
I20260928 20:23:29.136816 21261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ea0363ba4334acc838268facd720683 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ea0363ba4334acc838268facd720683" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ea0363ba4334acc838268facd720683" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 42701 } } }
I20260928 20:23:29.137045 21261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ea0363ba4334acc838268facd720683 [sys.catalog]: This master's current role is: LEADER
W20260928 20:23:29.138159 21275 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4ea0363ba4334acc838268facd720683: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 20:23:29.138223 21275 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 20:23:29.138280 21269 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:23:29.138450 21269 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:23:29.140293 21269 catalog_manager.cc:1347] Generated new cluster ID: 0897537018324e47b0248cfba988a819
I20260928 20:23:29.140348 21269 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 20:23:29 dist-test-slave-b83v krb5kdc[21174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:23:29.145119 21259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:29.134217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53158 (local address 127.3.18.62:42701)
0928 20:23:29.134424 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:29.134435 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:29.134472 (+ 37us) server_negotiation.cc:407] Connection header received
0928 20:23:29.134690 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:29.134698 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:29.134760 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:29.134912 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:29.135561 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:29.136992 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:29.138101 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:29.138718 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:29.138827 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:29.141449 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:29.141454 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:29.141454 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:23:29.141466 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:29.143206 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:29.143602 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:29.143606 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:29.143606 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:29.143711 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:29.143953 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:29.143954 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:29.143954 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:29.144211 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:29.144787 (+ 576us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:29.144845 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 20:23:29.144956 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":101,"threads_started":1}
I20260928 20:23:29.161398 21269 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:23:29.162247 21269 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:23:29.174993 21269 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ea0363ba4334acc838268facd720683: Generated new TSK 0
I20260928 20:23:29.175249 21269 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.18.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab.
Entry for principal kudu/127.3.18.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab.
I20260928 20:23:29.212461 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.1:0
--local_ip_for_outbound_sockets=127.3.18.1
--webserver_interface=127.3.18.1
--webserver_port=0
--tserver_master_addrs=127.3.18.62:42701
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab
--principal=kudu/127.3.18.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:38827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:23:29.297969 21281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:23:29.298185 21281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:23:29.298211 21281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:23:29.299369 21281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:23:29.299408 21281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:23:29.299435 21281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.1
I20260928 20:23:29.300915 21281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:38827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab
--principal=kudu/127.3.18.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.18.1
--webserver_port=0
--tserver_master_addrs=127.3.18.62:42701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.1
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:23:29.301143 21281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:23:29.301337 21281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:23:29.307106 21289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:29.307098 21287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:29.307094 21286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:23:29.307145 21281 server_base.cc:1034] running on GCE node
I20260928 20:23:29.307807 21281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:23:29.308156 21281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:23:29.309393 21281 hybrid_clock.cc:648] HybridClock initialized: now 1790627009309373 us; error 40 us; skew 500 ppm
Sep 28 20:23:29 dist-test-slave-b83v krb5kdc[21174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627009, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:23:29.312657 21281 init.cc:377] Logged in from keytab as kudu/127.3.18.1@KRBTEST.COM (short username kudu)
I20260928 20:23:29.313163 21281 webserver.cc:460] Webserver started at http://127.3.18.1:34635/ using document root <none> and password file <none>
I20260928 20:23:29.313432 21281 fs_manager.cc:359] Metadata directory not provided
I20260928 20:23:29.313481 21281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:23:29.313584 21281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:23:29.314786 21281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/minicluster-data/ts-0/data/instance:
uuid: "1055c5390d8e4b129748fccf1377f257"
format_stamp: "Formatted at 2026-09-28 20:23:29 on dist-test-slave-b83v"
server_key: "ac6228b366d5d1b25f739544c3fb0473"
server_key_iv: "7d49bf53ebe0f90ba82fceadcb7c9e88"
server_key_version: "encryptionkey@0"
I20260928 20:23:29.315158 21281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/minicluster-data/ts-0/wal/instance:
uuid: "1055c5390d8e4b129748fccf1377f257"
format_stamp: "Formatted at 2026-09-28 20:23:29 on dist-test-slave-b83v"
server_key: "ac6228b366d5d1b25f739544c3fb0473"
server_key_iv: "7d49bf53ebe0f90ba82fceadcb7c9e88"
server_key_version: "encryptionkey@0"
I20260928 20:23:29.316689 21281 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:23:29.317667 21296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:29.317993 21281 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:23:29.318138 21281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/minicluster-data/ts-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/minicluster-data/ts-0/wal
uuid: "1055c5390d8e4b129748fccf1377f257"
format_stamp: "Formatted at 2026-09-28 20:23:29 on dist-test-slave-b83v"
server_key: "ac6228b366d5d1b25f739544c3fb0473"
server_key_iv: "7d49bf53ebe0f90ba82fceadcb7c9e88"
server_key_version: "encryptionkey@0"
I20260928 20:23:29.318234 21281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:23:29.332020 21281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:23:29.334580 21281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:23:29.334805 21281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:23:29.335093 21281 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:23:29.335453 21281 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:23:29.335515 21281 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:29.335598 21281 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:23:29.335641 21281 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:29.342866 21281 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.1:35421
I20260928 20:23:29.342918 21409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.1:35421 every 8 connection(s)
I20260928 20:23:29.343290 21281 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
I20260928 20:23:29.348109 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 21281
I20260928 20:23:29.348304 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/minicluster-data/ts-0/wal/instance
I20260928 20:23:29.348506 3144 external_mini_cluster.cc:1428] Setting key 864802994cfffb987559bf6ee9d12e59
Sep 28 20:23:29 dist-test-slave-b83v krb5kdc[21174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627009, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.1@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:23:29.355057 21259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:29.344265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:49631 (local address 127.3.18.62:42701)
0928 20:23:29.344375 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:29.344378 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:29.344592 (+ 214us) server_negotiation.cc:407] Connection header received
0928 20:23:29.345266 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:29.345267 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:29.345284 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:29.345350 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:29.346448 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:29.347021 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:29.348452 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:29.348738 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:29.348829 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:29.351651 (+ 2822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:29.351653 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:29.351654 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:23:29.351663 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:29.353413 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:29.353966 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:29.353968 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:29.353969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:29.354035 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:29.354318 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:29.354319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:29.354320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:29.354451 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:29.354532 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:29.354843 (+ 311us) server_negotiation.cc:299] Negotiation successful
0928 20:23:29.354959 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 20:23:29.355291 21412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:29.344229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:42701 (local address 127.3.18.1:49631)
0928 20:23:29.344451 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:29.344497 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 20:23:29.345133 (+ 636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:23:29.345434 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:23:29.345438 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:23:29.346078 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:23:29.346317 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:29.346323 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:29.347149 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:29.347150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:29.348263 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:29.348266 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:29.348960 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:29.348961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:29.349149 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:29.349959 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:23:29.349969 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:23:29.351522 (+ 1553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:23:29.353598 (+ 2076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:23:29.353615 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:23:29.353619 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:23:29.353870 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:23:29.354106 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:23:29.354107 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:23:29.354108 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:23:29.354244 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:23:29.354524 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:23:29.354532 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:23:29.354700 (+ 168us) client_negotiation.cc:769] Sending connection context
0928 20:23:29.354817 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 20:23:29.355014 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":103,"threads_started":1}
I20260928 20:23:29.356266 21410 heartbeater.cc:347] Connected to a master server at 127.3.18.62:42701
I20260928 20:23:29.356374 21410 heartbeater.cc:464] Registering TS with master...
I20260928 20:23:29.356644 21410 heartbeater.cc:511] Master 127.3.18.62:42701 requested a full tablet report, sending...
I20260928 20:23:29.357348 21219 ts_manager.cc:194] Registered new tserver with Master: 1055c5390d8e4b129748fccf1377f257 (127.3.18.1:35421)
WARNING: no policy specified for kudu/127.3.18.2@KRBTEST.COM; defaulting to no policy
I20260928 20:23:29.358107 21219 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.18.1@KRBTEST.COM'} at 127.3.18.1:49631
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab.
Entry for principal kudu/127.3.18.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab.
I20260928 20:23:29.379498 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.2:0
--local_ip_for_outbound_sockets=127.3.18.2
--webserver_interface=127.3.18.2
--webserver_port=0
--tserver_master_addrs=127.3.18.62:42701
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab
--principal=kudu/127.3.18.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:38827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:23:29.463617 21415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:23:29.463884 21415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:23:29.463910 21415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:23:29.465178 21415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:23:29.465225 21415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:23:29.465252 21415 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.2
I20260928 20:23:29.466660 21415 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:38827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab
--principal=kudu/127.3.18.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.18.2
--webserver_port=0
--tserver_master_addrs=127.3.18.62:42701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.2
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:23:29.466864 21415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:23:29.467048 21415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:23:29.472332 21423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:23:29.472391 21415 server_base.cc:1034] running on GCE node
W20260928 20:23:29.472314 21421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:29.472311 21420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:23:29.472813 21415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:23:29.473076 21415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:23:29.474288 21415 hybrid_clock.cc:648] HybridClock initialized: now 1790627009474266 us; error 40 us; skew 500 ppm
Sep 28 20:23:29 dist-test-slave-b83v krb5kdc[21174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627009, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:23:29.477249 21415 init.cc:377] Logged in from keytab as kudu/127.3.18.2@KRBTEST.COM (short username kudu)
I20260928 20:23:29.477818 21415 webserver.cc:460] Webserver started at http://127.3.18.2:45773/ using document root <none> and password file <none>
I20260928 20:23:29.478080 21415 fs_manager.cc:359] Metadata directory not provided
I20260928 20:23:29.478163 21415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:23:29.478331 21415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:23:29.479513 21415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/minicluster-data/ts-1/data/instance:
uuid: "15e2823596f2463d8b75517562e7d44a"
format_stamp: "Formatted at 2026-09-28 20:23:29 on dist-test-slave-b83v"
server_key: "a7de45faca251dc4adb2e90b202bc8f9"
server_key_iv: "94c2295355cb5216a5199cfb29671a99"
server_key_version: "encryptionkey@0"
I20260928 20:23:29.479950 21415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/minicluster-data/ts-1/wal/instance:
uuid: "15e2823596f2463d8b75517562e7d44a"
format_stamp: "Formatted at 2026-09-28 20:23:29 on dist-test-slave-b83v"
server_key: "a7de45faca251dc4adb2e90b202bc8f9"
server_key_iv: "94c2295355cb5216a5199cfb29671a99"
server_key_version: "encryptionkey@0"
I20260928 20:23:29.481411 21415 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:23:29.482299 21430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:29.482543 21415 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 20:23:29.482712 21415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/minicluster-data/ts-1/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/minicluster-data/ts-1/wal
uuid: "15e2823596f2463d8b75517562e7d44a"
format_stamp: "Formatted at 2026-09-28 20:23:29 on dist-test-slave-b83v"
server_key: "a7de45faca251dc4adb2e90b202bc8f9"
server_key_iv: "94c2295355cb5216a5199cfb29671a99"
server_key_version: "encryptionkey@0"
I20260928 20:23:29.482800 21415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:23:29.506142 21415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:23:29.508862 21415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:23:29.509056 21415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:23:29.509330 21415 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:23:29.509668 21415 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:23:29.509727 21415 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:29.509771 21415 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:23:29.509796 21415 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:29.516425 21415 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.2:33303
I20260928 20:23:29.516467 21543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.2:33303 every 8 connection(s)
I20260928 20:23:29.516855 21415 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
Sep 28 20:23:29 dist-test-slave-b83v krb5kdc[21174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627009, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.2@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:23:29.525547 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 21415
I20260928 20:23:29.525791 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/minicluster-data/ts-1/wal/instance
I20260928 20:23:29.526065 3144 external_mini_cluster.cc:1428] Setting key 8df46fd0e00f37ee8798c3210a01e2d3
I20260928 20:23:29.527163 21259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:29.517714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:59159 (local address 127.3.18.62:42701)
0928 20:23:29.517796 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:29.517799 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:29.518064 (+ 265us) server_negotiation.cc:407] Connection header received
0928 20:23:29.518530 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:29.518532 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:29.518543 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:29.518625 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:29.519643 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:29.520249 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:29.521297 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:29.521642 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:29.521704 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:29.524106 (+ 2402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:29.524107 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:29.524107 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:23:29.524114 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:29.525526 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:29.526096 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:29.526097 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:29.526098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:29.526207 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:29.526496 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:29.526498 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:29.526498 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:29.526617 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:29.526694 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:29.526986 (+ 292us) server_negotiation.cc:299] Negotiation successful
0928 20:23:29.527069 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 20:23:29.527396 21546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:29.517647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:42701 (local address 127.3.18.2:59159)
0928 20:23:29.517865 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:29.517905 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 20:23:29.518433 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:23:29.518694 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:23:29.518698 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:23:29.519276 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:23:29.519477 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:29.519481 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:29.520347 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:29.520348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:29.521218 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:29.521221 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:29.521754 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:29.521755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:29.521871 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:29.522381 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:23:29.522391 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:23:29.523987 (+ 1596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:23:29.525617 (+ 1630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:23:29.525625 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:23:29.525628 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:23:29.525995 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:23:29.526287 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:23:29.526289 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:23:29.526289 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:23:29.526418 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:23:29.526686 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:23:29.526693 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:23:29.526847 (+ 154us) client_negotiation.cc:769] Sending connection context
0928 20:23:29.526955 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 20:23:29.527130 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":126,"threads_started":1}
I20260928 20:23:29.527900 21544 heartbeater.cc:347] Connected to a master server at 127.3.18.62:42701
I20260928 20:23:29.527997 21544 heartbeater.cc:464] Registering TS with master...
I20260928 20:23:29.528151 21544 heartbeater.cc:511] Master 127.3.18.62:42701 requested a full tablet report, sending...
I20260928 20:23:29.528537 21219 ts_manager.cc:194] Registered new tserver with Master: 15e2823596f2463d8b75517562e7d44a (127.3.18.2:33303)
I20260928 20:23:29.528916 21219 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.18.2@KRBTEST.COM'} at 127.3.18.2:59159
WARNING: no policy specified for kudu/127.3.18.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab.
Entry for principal kudu/127.3.18.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab.
I20260928 20:23:29.554924 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.3:0
--local_ip_for_outbound_sockets=127.3.18.3
--webserver_interface=127.3.18.3
--webserver_port=0
--tserver_master_addrs=127.3.18.62:42701
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab
--principal=kudu/127.3.18.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:38827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:23:29.636804 21549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:23:29.637084 21549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:23:29.637127 21549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:23:29.638564 21549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:23:29.638638 21549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:23:29.638685 21549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.3
I20260928 20:23:29.640235 21549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:38827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab
--principal=kudu/127.3.18.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.18.3
--webserver_port=0
--tserver_master_addrs=127.3.18.62:42701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.3
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:23:29.640504 21549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:23:29.640750 21549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:23:29.646281 21557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:29.646318 21555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:29.646296 21554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:23:29.646440 21549 server_base.cc:1034] running on GCE node
I20260928 20:23:29.646796 21549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:23:29.647080 21549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:23:29.648234 21549 hybrid_clock.cc:648] HybridClock initialized: now 1790627009648210 us; error 46 us; skew 500 ppm
Sep 28 20:23:29 dist-test-slave-b83v krb5kdc[21174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627009, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:23:29.651813 21549 init.cc:377] Logged in from keytab as kudu/127.3.18.3@KRBTEST.COM (short username kudu)
I20260928 20:23:29.652339 21549 webserver.cc:460] Webserver started at http://127.3.18.3:38823/ using document root <none> and password file <none>
I20260928 20:23:29.652623 21549 fs_manager.cc:359] Metadata directory not provided
I20260928 20:23:29.652702 21549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:23:29.652858 21549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:23:29.654037 21549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/minicluster-data/ts-2/data/instance:
uuid: "303a3f6f846a4286be3877eac098110f"
format_stamp: "Formatted at 2026-09-28 20:23:29 on dist-test-slave-b83v"
server_key: "ff96a3180feae309ced09ed4f7eeb8f4"
server_key_iv: "d0e792b3ce711998fe55330a0c945e84"
server_key_version: "encryptionkey@0"
I20260928 20:23:29.654448 21549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/minicluster-data/ts-2/wal/instance:
uuid: "303a3f6f846a4286be3877eac098110f"
format_stamp: "Formatted at 2026-09-28 20:23:29 on dist-test-slave-b83v"
server_key: "ff96a3180feae309ced09ed4f7eeb8f4"
server_key_iv: "d0e792b3ce711998fe55330a0c945e84"
server_key_version: "encryptionkey@0"
I20260928 20:23:29.655937 21549 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:23:29.656806 21564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:29.657002 21549 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:23:29.657111 21549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/minicluster-data/ts-2/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/minicluster-data/ts-2/wal
uuid: "303a3f6f846a4286be3877eac098110f"
format_stamp: "Formatted at 2026-09-28 20:23:29 on dist-test-slave-b83v"
server_key: "ff96a3180feae309ced09ed4f7eeb8f4"
server_key_iv: "d0e792b3ce711998fe55330a0c945e84"
server_key_version: "encryptionkey@0"
I20260928 20:23:29.657199 21549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:23:29.678234 21549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:23:29.680508 21549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:23:29.680693 21549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:23:29.680956 21549 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:23:29.681300 21549 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:23:29.681361 21549 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:29.681408 21549 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:23:29.681444 21549 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:29.687958 21549 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.3:38873
I20260928 20:23:29.688004 21677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.3:38873 every 8 connection(s)
I20260928 20:23:29.688398 21549 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
I20260928 20:23:29.690166 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 21549
I20260928 20:23:29.690294 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/minicluster-data/ts-2/wal/instance
I20260928 20:23:29.690471 3144 external_mini_cluster.cc:1428] Setting key d5bc893225c0c923e4fab4feddc492de
Sep 28 20:23:29 dist-test-slave-b83v krb5kdc[21174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627009, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.3@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:23:29.698701 21259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:29.689138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:38933 (local address 127.3.18.62:42701)
0928 20:23:29.689217 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:29.689219 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:29.689443 (+ 224us) server_negotiation.cc:407] Connection header received
0928 20:23:29.689914 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:29.689916 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:29.689927 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:29.690033 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:29.691380 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:29.692095 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:29.693142 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:29.693340 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:29.693402 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:29.695624 (+ 2222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:29.695625 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:29.695626 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:23:29.695635 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:29.697273 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:29.697762 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:29.697763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:29.697764 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:29.697817 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:29.698073 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:29.698075 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:29.698075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:29.698208 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:29.698282 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:29.698548 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 20:23:29.698632 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 20:23:29.698927 21680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:29.689151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:42701 (local address 127.3.18.3:38933)
0928 20:23:29.689301 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:29.689339 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 20:23:29.689824 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:23:29.690148 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:23:29.690152 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:23:29.690986 (+ 834us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:23:29.691288 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:29.691294 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:29.692181 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:29.692182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:29.693031 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:29.693033 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:29.693423 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:29.693424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:29.693534 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:29.694030 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:23:29.694038 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:23:29.695523 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:23:29.697357 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:23:29.697364 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:23:29.697365 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:23:29.697688 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:23:29.697881 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:23:29.697881 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:23:29.697882 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:23:29.697985 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:23:29.698282 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:23:29.698286 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:23:29.698412 (+ 126us) client_negotiation.cc:769] Sending connection context
0928 20:23:29.698503 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 20:23:29.698645 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":82,"threads_started":1}
I20260928 20:23:29.699498 21678 heartbeater.cc:347] Connected to a master server at 127.3.18.62:42701
I20260928 20:23:29.699600 21678 heartbeater.cc:464] Registering TS with master...
I20260928 20:23:29.699751 21678 heartbeater.cc:511] Master 127.3.18.62:42701 requested a full tablet report, sending...
I20260928 20:23:29.700212 21219 ts_manager.cc:194] Registered new tserver with Master: 303a3f6f846a4286be3877eac098110f (127.3.18.3:38873)
I20260928 20:23:29.700613 21219 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.18.3@KRBTEST.COM'} at 127.3.18.3:38933
I20260928 20:23:29.702553 3144 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 20:23:29.710486 21259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:29.704002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53162 (local address 127.3.18.62:42701)
0928 20:23:29.704045 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:29.704047 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:29.704192 (+ 145us) server_negotiation.cc:407] Connection header received
0928 20:23:29.704262 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:29.704263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:29.704271 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:29.704322 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:29.704882 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:29.705398 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:29.706193 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:29.706391 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:29.706447 (+ 56us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:29.708029 (+ 1582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:29.708030 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:29.708030 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:23:29.708037 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:29.709365 (+ 1328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:29.709769 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:29.709770 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:29.709770 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:29.709825 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:29.710022 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:29.710022 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:29.710022 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:29.710163 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:29.710249 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:29.710366 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 20:23:29.710417 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 20:23:29.712455 21219 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53162:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 20:23:29.714247 21219 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 20:23:29.721379 21693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:29.717252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47780 (local address 127.3.18.1:35421)
0928 20:23:29.717405 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:29.717410 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:29.717420 (+ 10us) server_negotiation.cc:407] Connection header received
0928 20:23:29.717458 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:29.717460 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:29.717476 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:29.717569 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:23:29.717751 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:29.718919 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:29.720163 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:29.720814 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:29.720905 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:29.720966 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:29.721218 (+ 252us) server_negotiation.cc:299] Negotiation successful
0928 20:23:29.721269 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":76,"threads_started":1}
I20260928 20:23:29.721385 21689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:29.716940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.1:35421 (local address 127.0.0.1:47780)
0928 20:23:29.717103 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:29.717146 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 20:23:29.717217 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:23:29.717585 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:23:29.717588 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:23:29.717590 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:23:29.717664 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:29.717666 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:29.719038 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:29.719039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:29.720080 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:29.720083 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:29.720944 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:29.720946 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:29.721032 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:29.721045 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 20:23:29.721200 (+ 155us) client_negotiation.cc:240] Negotiation successful
0928 20:23:29.721286 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":50,"threads_started":1}
I20260928 20:23:29.721963 21690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:29.716953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.2:33303 (local address 127.0.0.1:59964)
0928 20:23:29.717130 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:29.717150 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 20:23:29.717217 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:23:29.717880 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:23:29.717881 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:23:29.717883 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:23:29.717927 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:29.717928 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:29.719195 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:29.719196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:29.720300 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:29.720302 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:29.721269 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:29.721270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:29.721327 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:29.721331 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:23:29.721392 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 20:23:29.721874 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":120,"threads_started":1}
I20260928 20:23:29.722508 21692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:29.717226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59964 (local address 127.3.18.2:33303)
0928 20:23:29.717712 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:29.717717 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:29.717727 (+ 10us) server_negotiation.cc:407] Connection header received
0928 20:23:29.717763 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:29.717765 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:29.717782 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:29.717998 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:23:29.718088 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:29.719094 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:29.720414 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:29.721182 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:29.721255 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:29.721308 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:29.722371 (+ 1063us) server_negotiation.cc:299] Negotiation successful
0928 20:23:29.722421 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":99,"threads_started":1}
I20260928 20:23:29.722301 21344 tablet_service.cc:1467] Processing CreateTablet for tablet bbaf0b91b31b49a5912411fa716e255a (DEFAULT_TABLE table=test-table [id=6e244fad9b314bdfa0ce08eafb019e6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:23:29.722522 21691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:29.716955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.3:38873 (local address 127.0.0.1:42934)
0928 20:23:29.717318 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:29.717327 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 20:23:29.717366 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:23:29.717926 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:23:29.717927 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:23:29.717929 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:23:29.717988 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:29.717990 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:29.718918 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:29.718919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:29.720272 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:29.720275 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:29.722248 (+ 1973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:29.722249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:29.722310 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:29.722314 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:23:29.722408 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 20:23:29.722454 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"mutex_wait_us":37,"thread_start_us":144,"threads_started":1}
I20260928 20:23:29.722546 21694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:29.717413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42934 (local address 127.3.18.3:38873)
0928 20:23:29.717591 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:29.717595 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:29.717608 (+ 13us) server_negotiation.cc:407] Connection header received
0928 20:23:29.717733 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:29.717736 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:29.717754 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:29.717835 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:23:29.718071 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:29.718799 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:29.721346 (+ 2547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:29.722085 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:29.722153 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:29.722212 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:29.722428 (+ 216us) server_negotiation.cc:299] Negotiation successful
0928 20:23:29.722470 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":102,"threads_started":1}
I20260928 20:23:29.722801 21344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbaf0b91b31b49a5912411fa716e255a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:23:29.723856 21612 tablet_service.cc:1467] Processing CreateTablet for tablet bbaf0b91b31b49a5912411fa716e255a (DEFAULT_TABLE table=test-table [id=6e244fad9b314bdfa0ce08eafb019e6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:23:29.724050 21478 tablet_service.cc:1467] Processing CreateTablet for tablet bbaf0b91b31b49a5912411fa716e255a (DEFAULT_TABLE table=test-table [id=6e244fad9b314bdfa0ce08eafb019e6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:23:29.724246 21612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbaf0b91b31b49a5912411fa716e255a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:23:29.724400 21478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbaf0b91b31b49a5912411fa716e255a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:23:29.726583 21695 tablet_bootstrap.cc:492] T bbaf0b91b31b49a5912411fa716e255a P 1055c5390d8e4b129748fccf1377f257: Bootstrap starting.
I20260928 20:23:29.727408 21695 tablet_bootstrap.cc:654] T bbaf0b91b31b49a5912411fa716e255a P 1055c5390d8e4b129748fccf1377f257: Neither blocks nor log segments found. Creating new log.
I20260928 20:23:29.727694 21696 tablet_bootstrap.cc:492] T bbaf0b91b31b49a5912411fa716e255a P 303a3f6f846a4286be3877eac098110f: Bootstrap starting.
I20260928 20:23:29.727906 21697 tablet_bootstrap.cc:492] T bbaf0b91b31b49a5912411fa716e255a P 15e2823596f2463d8b75517562e7d44a: Bootstrap starting.
I20260928 20:23:29.728135 21695 log.cc:824] T bbaf0b91b31b49a5912411fa716e255a P 1055c5390d8e4b129748fccf1377f257: Log is configured to *not* fsync() on all Append() calls
I20260928 20:23:29.728642 21697 tablet_bootstrap.cc:654] T bbaf0b91b31b49a5912411fa716e255a P 15e2823596f2463d8b75517562e7d44a: Neither blocks nor log segments found. Creating new log.
I20260928 20:23:29.728958 21695 tablet_bootstrap.cc:492] T bbaf0b91b31b49a5912411fa716e255a P 1055c5390d8e4b129748fccf1377f257: No bootstrap required, opened a new log
I20260928 20:23:29.729017 21695 ts_tablet_manager.cc:1397] T bbaf0b91b31b49a5912411fa716e255a P 1055c5390d8e4b129748fccf1377f257: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 20:23:29.729220 21697 log.cc:824] T bbaf0b91b31b49a5912411fa716e255a P 15e2823596f2463d8b75517562e7d44a: Log is configured to *not* fsync() on all Append() calls
I20260928 20:23:29.730088 21697 tablet_bootstrap.cc:492] T bbaf0b91b31b49a5912411fa716e255a P 15e2823596f2463d8b75517562e7d44a: No bootstrap required, opened a new log
I20260928 20:23:29.730091 21696 tablet_bootstrap.cc:654] T bbaf0b91b31b49a5912411fa716e255a P 303a3f6f846a4286be3877eac098110f: Neither blocks nor log segments found. Creating new log.
I20260928 20:23:29.730171 21697 ts_tablet_manager.cc:1397] T bbaf0b91b31b49a5912411fa716e255a P 15e2823596f2463d8b75517562e7d44a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 20:23:29.730625 21696 log.cc:824] T bbaf0b91b31b49a5912411fa716e255a P 303a3f6f846a4286be3877eac098110f: Log is configured to *not* fsync() on all Append() calls
I20260928 20:23:29.730993 21695 raft_consensus.cc:348] T bbaf0b91b31b49a5912411fa716e255a P 1055c5390d8e4b129748fccf1377f257 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e2823596f2463d8b75517562e7d44a" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 33303 } } peers { permanent_uuid: "1055c5390d8e4b129748fccf1377f257" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 35421 } } peers { permanent_uuid: "303a3f6f846a4286be3877eac098110f" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38873 } }
I20260928 20:23:29.731218 21695 raft_consensus.cc:374] T bbaf0b91b31b49a5912411fa716e255a P 1055c5390d8e4b129748fccf1377f257 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:23:29.731271 21695 raft_consensus.cc:729] T bbaf0b91b31b49a5912411fa716e255a P 1055c5390d8e4b129748fccf1377f257 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1055c5390d8e4b129748fccf1377f257, State: Initialized, Role: FOLLOWER
I20260928 20:23:29.731406 21696 tablet_bootstrap.cc:492] T bbaf0b91b31b49a5912411fa716e255a P 303a3f6f846a4286be3877eac098110f: No bootstrap required, opened a new log
I20260928 20:23:29.731432 21695 consensus_queue.cc:260] T bbaf0b91b31b49a5912411fa716e255a P 1055c5390d8e4b129748fccf1377f257 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e2823596f2463d8b75517562e7d44a" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 33303 } } peers { permanent_uuid: "1055c5390d8e4b129748fccf1377f257" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 35421 } } peers { permanent_uuid: "303a3f6f846a4286be3877eac098110f" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38873 } }
I20260928 20:23:29.731508 21696 ts_tablet_manager.cc:1397] T bbaf0b91b31b49a5912411fa716e255a P 303a3f6f846a4286be3877eac098110f: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260928 20:23:29.731787 21695 ts_tablet_manager.cc:1428] T bbaf0b91b31b49a5912411fa716e255a P 1055c5390d8e4b129748fccf1377f257: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 20:23:29.731916 21410 heartbeater.cc:503] Master 127.3.18.62:42701 was elected leader, sending a full tablet report...
I20260928 20:23:29.732414 21697 raft_consensus.cc:348] T bbaf0b91b31b49a5912411fa716e255a P 15e2823596f2463d8b75517562e7d44a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e2823596f2463d8b75517562e7d44a" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 33303 } } peers { permanent_uuid: "1055c5390d8e4b129748fccf1377f257" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 35421 } } peers { permanent_uuid: "303a3f6f846a4286be3877eac098110f" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38873 } }
I20260928 20:23:29.732724 21697 raft_consensus.cc:374] T bbaf0b91b31b49a5912411fa716e255a P 15e2823596f2463d8b75517562e7d44a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:23:29.732793 21697 raft_consensus.cc:729] T bbaf0b91b31b49a5912411fa716e255a P 15e2823596f2463d8b75517562e7d44a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15e2823596f2463d8b75517562e7d44a, State: Initialized, Role: FOLLOWER
I20260928 20:23:29.732928 21697 consensus_queue.cc:260] T bbaf0b91b31b49a5912411fa716e255a P 15e2823596f2463d8b75517562e7d44a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e2823596f2463d8b75517562e7d44a" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 33303 } } peers { permanent_uuid: "1055c5390d8e4b129748fccf1377f257" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 35421 } } peers { permanent_uuid: "303a3f6f846a4286be3877eac098110f" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38873 } }
I20260928 20:23:29.733275 21544 heartbeater.cc:503] Master 127.3.18.62:42701 was elected leader, sending a full tablet report...
I20260928 20:23:29.733273 21697 ts_tablet_manager.cc:1428] T bbaf0b91b31b49a5912411fa716e255a P 15e2823596f2463d8b75517562e7d44a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 20:23:29.733525 21696 raft_consensus.cc:348] T bbaf0b91b31b49a5912411fa716e255a P 303a3f6f846a4286be3877eac098110f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e2823596f2463d8b75517562e7d44a" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 33303 } } peers { permanent_uuid: "1055c5390d8e4b129748fccf1377f257" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 35421 } } peers { permanent_uuid: "303a3f6f846a4286be3877eac098110f" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38873 } }
I20260928 20:23:29.733712 21696 raft_consensus.cc:374] T bbaf0b91b31b49a5912411fa716e255a P 303a3f6f846a4286be3877eac098110f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:23:29.733768 21696 raft_consensus.cc:729] T bbaf0b91b31b49a5912411fa716e255a P 303a3f6f846a4286be3877eac098110f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 303a3f6f846a4286be3877eac098110f, State: Initialized, Role: FOLLOWER
I20260928 20:23:29.733919 21696 consensus_queue.cc:260] T bbaf0b91b31b49a5912411fa716e255a P 303a3f6f846a4286be3877eac098110f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e2823596f2463d8b75517562e7d44a" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 33303 } } peers { permanent_uuid: "1055c5390d8e4b129748fccf1377f257" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 35421 } } peers { permanent_uuid: "303a3f6f846a4286be3877eac098110f" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38873 } }
I20260928 20:23:29.734313 21696 ts_tablet_manager.cc:1428] T bbaf0b91b31b49a5912411fa716e255a P 303a3f6f846a4286be3877eac098110f: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260928 20:23:29.734390 21678 heartbeater.cc:503] Master 127.3.18.62:42701 was elected leader, sending a full tablet report...
W20260928 20:23:29.767467 21545 tablet.cc:2369] T bbaf0b91b31b49a5912411fa716e255a P 15e2823596f2463d8b75517562e7d44a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 20:23:29.844059 21411 tablet.cc:2369] T bbaf0b91b31b49a5912411fa716e255a P 1055c5390d8e4b129748fccf1377f257: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 20:23:29.858633 21703 raft_consensus.cc:482] T bbaf0b91b31b49a5912411fa716e255a P 303a3f6f846a4286be3877eac098110f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:23:29.858768 21703 raft_consensus.cc:504] T bbaf0b91b31b49a5912411fa716e255a P 303a3f6f846a4286be3877eac098110f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e2823596f2463d8b75517562e7d44a" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 33303 } } peers { permanent_uuid: "1055c5390d8e4b129748fccf1377f257" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 35421 } } peers { permanent_uuid: "303a3f6f846a4286be3877eac098110f" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38873 } }
I20260928 20:23:29.859122 21703 leader_election.cc:288] T bbaf0b91b31b49a5912411fa716e255a P 303a3f6f846a4286be3877eac098110f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15e2823596f2463d8b75517562e7d44a (127.3.18.2:33303), 1055c5390d8e4b129748fccf1377f257 (127.3.18.1:35421)
I20260928 20:23:29.863664 21693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:29.859518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:44705 (local address 127.3.18.1:35421)
0928 20:23:29.859568 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:29.859570 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:29.859735 (+ 165us) server_negotiation.cc:407] Connection header received
0928 20:23:29.859862 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:29.859863 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:29.859895 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:29.859954 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:23:29.860183 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:29.860994 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:29.862664 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:29.863272 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:29.863361 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:29.863372 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:29.863554 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 20:23:29.863605 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 20:23:29.863694 21680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:29.859355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.2:33303 (local address 127.3.18.3:39225)
0928 20:23:29.859438 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:29.859451 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 20:23:29.859524 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:23:29.859962 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:23:29.859963 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:23:29.859965 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:23:29.860069 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:29.860071 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:29.861162 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:29.861163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:29.862486 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:29.862489 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:29.863373 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:29.863374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:29.863460 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:29.863461 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 20:23:29.863525 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 20:23:29.863625 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":40}
I20260928 20:23:29.863695 21704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:29.859364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.1:35421 (local address 127.3.18.3:44705)
0928 20:23:29.859634 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:29.859643 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 20:23:29.859731 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:23:29.859962 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:23:29.859963 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:23:29.859965 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:23:29.860076 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:29.860078 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:29.861094 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:29.861095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:29.862486 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:29.862489 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:29.863373 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:29.863374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:29.863456 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:29.863461 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 20:23:29.863525 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 20:23:29.863576 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":146,"thread_start_us":117,"threads_started":1}
I20260928 20:23:29.864094 21692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:29.859643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:39225 (local address 127.3.18.2:33303)
0928 20:23:29.859800 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:29.859802 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:29.859814 (+ 12us) server_negotiation.cc:407] Connection header received
0928 20:23:29.859850 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:29.859851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:29.859867 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:29.859952 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:23:29.860183 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:29.861086 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:29.862598 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:29.863270 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:29.863361 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:29.863372 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:29.863949 (+ 577us) server_negotiation.cc:299] Negotiation successful
0928 20:23:29.864009 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260928 20:23:29.864306 21364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbaf0b91b31b49a5912411fa716e255a" candidate_uuid: "303a3f6f846a4286be3877eac098110f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1055c5390d8e4b129748fccf1377f257" is_pre_election: true
I20260928 20:23:29.864475 21498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbaf0b91b31b49a5912411fa716e255a" candidate_uuid: "303a3f6f846a4286be3877eac098110f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15e2823596f2463d8b75517562e7d44a" is_pre_election: true
I20260928 20:23:29.864565 21364 raft_consensus.cc:2454] T bbaf0b91b31b49a5912411fa716e255a P 1055c5390d8e4b129748fccf1377f257 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 303a3f6f846a4286be3877eac098110f in term 0.
I20260928 20:23:29.864841 21498 raft_consensus.cc:2454] T bbaf0b91b31b49a5912411fa716e255a P 15e2823596f2463d8b75517562e7d44a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 303a3f6f846a4286be3877eac098110f in term 0.
I20260928 20:23:29.864866 21565 leader_election.cc:302] T bbaf0b91b31b49a5912411fa716e255a P 303a3f6f846a4286be3877eac098110f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1055c5390d8e4b129748fccf1377f257, 303a3f6f846a4286be3877eac098110f; no voters:
I20260928 20:23:29.865011 21703 raft_consensus.cc:2781] T bbaf0b91b31b49a5912411fa716e255a P 303a3f6f846a4286be3877eac098110f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 20:23:29.865110 21703 raft_consensus.cc:482] T bbaf0b91b31b49a5912411fa716e255a P 303a3f6f846a4286be3877eac098110f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 20:23:29.865147 21703 raft_consensus.cc:3037] T bbaf0b91b31b49a5912411fa716e255a P 303a3f6f846a4286be3877eac098110f [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:23:29.865866 21703 raft_consensus.cc:504] T bbaf0b91b31b49a5912411fa716e255a P 303a3f6f846a4286be3877eac098110f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e2823596f2463d8b75517562e7d44a" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 33303 } } peers { permanent_uuid: "1055c5390d8e4b129748fccf1377f257" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 35421 } } peers { permanent_uuid: "303a3f6f846a4286be3877eac098110f" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38873 } }
I20260928 20:23:29.866051 21703 leader_election.cc:288] T bbaf0b91b31b49a5912411fa716e255a P 303a3f6f846a4286be3877eac098110f [CANDIDATE]: Term 1 election: Requested vote from peers 15e2823596f2463d8b75517562e7d44a (127.3.18.2:33303), 1055c5390d8e4b129748fccf1377f257 (127.3.18.1:35421)
I20260928 20:23:29.866374 21364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbaf0b91b31b49a5912411fa716e255a" candidate_uuid: "303a3f6f846a4286be3877eac098110f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1055c5390d8e4b129748fccf1377f257"
I20260928 20:23:29.866374 21498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbaf0b91b31b49a5912411fa716e255a" candidate_uuid: "303a3f6f846a4286be3877eac098110f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15e2823596f2463d8b75517562e7d44a"
I20260928 20:23:29.866480 21364 raft_consensus.cc:3037] T bbaf0b91b31b49a5912411fa716e255a P 1055c5390d8e4b129748fccf1377f257 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:23:29.866533 21498 raft_consensus.cc:3037] T bbaf0b91b31b49a5912411fa716e255a P 15e2823596f2463d8b75517562e7d44a [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:23:29.867449 21364 raft_consensus.cc:2454] T bbaf0b91b31b49a5912411fa716e255a P 1055c5390d8e4b129748fccf1377f257 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 303a3f6f846a4286be3877eac098110f in term 1.
I20260928 20:23:29.867456 21498 raft_consensus.cc:2454] T bbaf0b91b31b49a5912411fa716e255a P 15e2823596f2463d8b75517562e7d44a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 303a3f6f846a4286be3877eac098110f in term 1.
I20260928 20:23:29.867730 21565 leader_election.cc:302] T bbaf0b91b31b49a5912411fa716e255a P 303a3f6f846a4286be3877eac098110f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1055c5390d8e4b129748fccf1377f257, 303a3f6f846a4286be3877eac098110f; no voters:
I20260928 20:23:29.867892 21703 raft_consensus.cc:2781] T bbaf0b91b31b49a5912411fa716e255a P 303a3f6f846a4286be3877eac098110f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:23:29.868085 21703 raft_consensus.cc:686] T bbaf0b91b31b49a5912411fa716e255a P 303a3f6f846a4286be3877eac098110f [term 1 LEADER]: Becoming Leader. State: Replica: 303a3f6f846a4286be3877eac098110f, State: Running, Role: LEADER
I20260928 20:23:29.868257 21703 consensus_queue.cc:237] T bbaf0b91b31b49a5912411fa716e255a P 303a3f6f846a4286be3877eac098110f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e2823596f2463d8b75517562e7d44a" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 33303 } } peers { permanent_uuid: "1055c5390d8e4b129748fccf1377f257" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 35421 } } peers { permanent_uuid: "303a3f6f846a4286be3877eac098110f" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38873 } }
I20260928 20:23:29.869225 21218 catalog_manager.cc:5495] T bbaf0b91b31b49a5912411fa716e255a P 303a3f6f846a4286be3877eac098110f reported cstate change: term changed from 0 to 1, leader changed from <none> to 303a3f6f846a4286be3877eac098110f (127.3.18.3). New cstate: current_term: 1 leader_uuid: "303a3f6f846a4286be3877eac098110f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e2823596f2463d8b75517562e7d44a" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 33303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1055c5390d8e4b129748fccf1377f257" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 35421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "303a3f6f846a4286be3877eac098110f" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 38873 } health_report { overall_health: HEALTHY } } }
I20260928 20:23:29.905229 21694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:29.902614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42940 (local address 127.3.18.3:38873)
0928 20:23:29.902680 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:29.902682 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:29.902689 (+ 7us) server_negotiation.cc:407] Connection header received
0928 20:23:29.902726 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:29.902727 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:29.902747 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:29.902812 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:23:29.902950 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:29.903540 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:29.904411 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:29.904601 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:29.904660 (+ 59us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:29.904881 (+ 221us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:23:29.904969 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:23:29.905052 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:29.905157 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 20:23:29.905166 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 20:23:29.907056 21364 raft_consensus.cc:1261] T bbaf0b91b31b49a5912411fa716e255a P 1055c5390d8e4b129748fccf1377f257 [term 1 FOLLOWER]: Refusing update from remote peer 303a3f6f846a4286be3877eac098110f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:23:29.907056 21498 raft_consensus.cc:1261] T bbaf0b91b31b49a5912411fa716e255a P 15e2823596f2463d8b75517562e7d44a [term 1 FOLLOWER]: Refusing update from remote peer 303a3f6f846a4286be3877eac098110f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:23:29.907438 21703 consensus_queue.cc:1035] T bbaf0b91b31b49a5912411fa716e255a P 303a3f6f846a4286be3877eac098110f [LEADER]: Connected to new peer: Peer: permanent_uuid: "15e2823596f2463d8b75517562e7d44a" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 33303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:23:29.907648 21705 consensus_queue.cc:1035] T bbaf0b91b31b49a5912411fa716e255a P 303a3f6f846a4286be3877eac098110f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1055c5390d8e4b129748fccf1377f257" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 35421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:23:29.910270 21708 mvcc.cc:205] Tried to move back new op lower bound from 7334408232576475136 to 7334408232421175296. Current Snapshot: MvccSnapshot[applied={T|T < 7334408232576475136}]
I20260928 20:23:29.910604 21710 mvcc.cc:205] Tried to move back new op lower bound from 7334408232576475136 to 7334408232421175296. Current Snapshot: MvccSnapshot[applied={T|T < 7334408232576475136}]
I20260928 20:23:29.910710 21709 mvcc.cc:205] Tried to move back new op lower bound from 7334408232576475136 to 7334408232421175296. Current Snapshot: MvccSnapshot[applied={T|T < 7334408232576475136}]
I20260928 20:23:29.913647 21219 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:23:29.913756 21219 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:23:29.914516 21219 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4ea0363ba4334acc838268facd720683: Sending DeleteTablet for 3 replicas of tablet bbaf0b91b31b49a5912411fa716e255a
I20260928 20:23:29.914878 21478 tablet_service.cc:1514] Processing DeleteTablet for tablet bbaf0b91b31b49a5912411fa716e255a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:23:29 UTC) from {username='kudu', principal='kudu/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:59964
I20260928 20:23:29.914888 21344 tablet_service.cc:1514] Processing DeleteTablet for tablet bbaf0b91b31b49a5912411fa716e255a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:23:29 UTC) from {username='kudu', principal='kudu/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:47780
I20260928 20:23:29.915211 21719 tablet_replica.cc:330] T bbaf0b91b31b49a5912411fa716e255a P 1055c5390d8e4b129748fccf1377f257: stopping tablet replica
I20260928 20:23:29.915235 21612 tablet_service.cc:1514] Processing DeleteTablet for tablet bbaf0b91b31b49a5912411fa716e255a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:23:29 UTC) from {username='kudu', principal='kudu/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:42934
I20260928 20:23:29.915259 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 21281
I20260928 20:23:29.915416 21719 raft_consensus.cc:2229] T bbaf0b91b31b49a5912411fa716e255a P 1055c5390d8e4b129748fccf1377f257 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:23:29.915568 21719 raft_consensus.cc:2258] T bbaf0b91b31b49a5912411fa716e255a P 1055c5390d8e4b129748fccf1377f257 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:23:29.915798 21718 tablet_replica.cc:330] T bbaf0b91b31b49a5912411fa716e255a P 15e2823596f2463d8b75517562e7d44a: stopping tablet replica
I20260928 20:23:29.916117 21720 tablet_replica.cc:330] T bbaf0b91b31b49a5912411fa716e255a P 303a3f6f846a4286be3877eac098110f: stopping tablet replica
I20260928 20:23:29.916205 21720 raft_consensus.cc:2229] T bbaf0b91b31b49a5912411fa716e255a P 303a3f6f846a4286be3877eac098110f [term 1 LEADER]: Raft consensus shutting down.
I20260928 20:23:29.916316 21720 raft_consensus.cc:2258] T bbaf0b91b31b49a5912411fa716e255a P 303a3f6f846a4286be3877eac098110f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:23:29.916482 21718 raft_consensus.cc:2229] T bbaf0b91b31b49a5912411fa716e255a P 15e2823596f2463d8b75517562e7d44a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:23:29.916604 21718 raft_consensus.cc:2258] T bbaf0b91b31b49a5912411fa716e255a P 15e2823596f2463d8b75517562e7d44a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:23:29.917008 21718 ts_tablet_manager.cc:1905] T bbaf0b91b31b49a5912411fa716e255a P 15e2823596f2463d8b75517562e7d44a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:23:29.918081 21720 ts_tablet_manager.cc:1905] T bbaf0b91b31b49a5912411fa716e255a P 303a3f6f846a4286be3877eac098110f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:23:29.918756 21718 ts_tablet_manager.cc:1918] T bbaf0b91b31b49a5912411fa716e255a P 15e2823596f2463d8b75517562e7d44a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:23:29.918835 21718 log.cc:1196] T bbaf0b91b31b49a5912411fa716e255a P 15e2823596f2463d8b75517562e7d44a: Deleting WAL directory at /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/minicluster-data/ts-1/wal/wals/bbaf0b91b31b49a5912411fa716e255a
I20260928 20:23:29.919123 21718 ts_tablet_manager.cc:1939] T bbaf0b91b31b49a5912411fa716e255a P 15e2823596f2463d8b75517562e7d44a: Deleting consensus metadata
I20260928 20:23:29.919660 21206 catalog_manager.cc:4841] TS 15e2823596f2463d8b75517562e7d44a (127.3.18.2:33303): tablet bbaf0b91b31b49a5912411fa716e255a (table test-table [id=6e244fad9b314bdfa0ce08eafb019e6a]) successfully deleted
I20260928 20:23:29.920116 21720 ts_tablet_manager.cc:1918] T bbaf0b91b31b49a5912411fa716e255a P 303a3f6f846a4286be3877eac098110f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:23:29.920208 21720 log.cc:1196] T bbaf0b91b31b49a5912411fa716e255a P 303a3f6f846a4286be3877eac098110f: Deleting WAL directory at /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790626764250411-3144-0/minicluster-data/ts-2/wal/wals/bbaf0b91b31b49a5912411fa716e255a
I20260928 20:23:29.920452 21720 ts_tablet_manager.cc:1939] T bbaf0b91b31b49a5912411fa716e255a P 303a3f6f846a4286be3877eac098110f: Deleting consensus metadata
I20260928 20:23:29.920855 21205 catalog_manager.cc:4841] TS 303a3f6f846a4286be3877eac098110f (127.3.18.3:38873): tablet bbaf0b91b31b49a5912411fa716e255a (table test-table [id=6e244fad9b314bdfa0ce08eafb019e6a]) successfully deleted
W20260928 20:23:29.923740 21203 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.18.1:35421 (error 108)
I20260928 20:23:29.923904 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 21415
I20260928 20:23:29.930227 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 21549
I20260928 20:23:29.930387 21691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:29.930199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.1:35421 (local address 127.0.0.1:47784)
0928 20:23:29.930273 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:29.930332 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.1:35421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 20:23:29.931732 21203 catalog_manager.cc:4584] TS 1055c5390d8e4b129748fccf1377f257 (127.3.18.1:35421): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bbaf0b91b31b49a5912411fa716e255a: Network error: Client connection negotiation failed: client connection to 127.3.18.1:35421: connect: Connection refused (error 111)
I20260928 20:23:29.935165 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 21187
2026-09-28T20:23:29Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2096 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T20:23:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:23:29Z Disabled control of system clock
I20260928 20:23:29.965435 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790626764250411-3144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790626764250411-3144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790626764250411-3144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:34549
--webserver_interface=127.3.18.62
--webserver_port=0
--builtin_ntp_servers=127.3.18.20:37567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.18.62:34549
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 20:23:30.048563 21725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:23:30.048801 21725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:23:30.048823 21725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:23:30.049928 21725 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:23:30.049963 21725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:23:30.049976 21725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:23:30.049988 21725 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:23:30.050006 21725 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:23:30.051316 21725 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:37567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790626764250411-3144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790626764250411-3144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.18.62:34549
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:34549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.18.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790626764250411-3144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:23:30.051523 21725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:23:30.051800 21725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:23:30.056789 21730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:23:30.056936 21725 server_base.cc:1034] running on GCE node
W20260928 20:23:30.056808 21733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:30.056808 21731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:23:30.057387 21725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:23:30.057739 21725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:23:30.058903 21725 hybrid_clock.cc:648] HybridClock initialized: now 1790627010058888 us; error 32 us; skew 500 ppm
I20260928 20:23:30.060420 21725 webserver.cc:460] Webserver started at http://127.3.18.62:33379/ using document root <none> and password file <none>
I20260928 20:23:30.060736 21725 fs_manager.cc:359] Metadata directory not provided
I20260928 20:23:30.060796 21725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:23:30.060892 21725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:23:30.062029 21725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790626764250411-3144-0/minicluster-data/master-0/data/instance:
uuid: "2f95ded2ddf74ec88a694a3c96c6c9f1"
format_stamp: "Formatted at 2026-09-28 20:23:30 on dist-test-slave-b83v"
server_key: "4b573b36ae09a50682fd3a1361ad9cfc"
server_key_iv: "bee71b4c13d59a47bc3a4e0d95037d03"
server_key_version: "encryptionkey@0"
I20260928 20:23:30.062412 21725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790626764250411-3144-0/minicluster-data/master-0/wal/instance:
uuid: "2f95ded2ddf74ec88a694a3c96c6c9f1"
format_stamp: "Formatted at 2026-09-28 20:23:30 on dist-test-slave-b83v"
server_key: "4b573b36ae09a50682fd3a1361ad9cfc"
server_key_iv: "bee71b4c13d59a47bc3a4e0d95037d03"
server_key_version: "encryptionkey@0"
I20260928 20:23:30.063912 21725 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260928 20:23:30.064777 21739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:30.064975 21725 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:23:30.065080 21725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790626764250411-3144-0/minicluster-data/master-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790626764250411-3144-0/minicluster-data/master-0/wal
uuid: "2f95ded2ddf74ec88a694a3c96c6c9f1"
format_stamp: "Formatted at 2026-09-28 20:23:30 on dist-test-slave-b83v"
server_key: "4b573b36ae09a50682fd3a1361ad9cfc"
server_key_iv: "bee71b4c13d59a47bc3a4e0d95037d03"
server_key_version: "encryptionkey@0"
I20260928 20:23:30.065171 21725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790626764250411-3144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790626764250411-3144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790626764250411-3144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:23:30.075127 21725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:23:30.075493 21725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:23:30.075659 21725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:23:30.080014 21725 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.62:34549
I20260928 20:23:30.080049 21791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.62:34549 every 8 connection(s)
I20260928 20:23:30.080529 21725 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
I20260928 20:23:30.081688 21792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:23:30.084477 21792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f95ded2ddf74ec88a694a3c96c6c9f1: Bootstrap starting.
I20260928 20:23:30.085093 21792 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f95ded2ddf74ec88a694a3c96c6c9f1: Neither blocks nor log segments found. Creating new log.
I20260928 20:23:30.085378 21792 log.cc:824] T 00000000000000000000000000000000 P 2f95ded2ddf74ec88a694a3c96c6c9f1: Log is configured to *not* fsync() on all Append() calls
I20260928 20:23:30.086159 21792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f95ded2ddf74ec88a694a3c96c6c9f1: No bootstrap required, opened a new log
I20260928 20:23:30.088050 21792 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f95ded2ddf74ec88a694a3c96c6c9f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f95ded2ddf74ec88a694a3c96c6c9f1" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 34549 } }
I20260928 20:23:30.088227 21792 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f95ded2ddf74ec88a694a3c96c6c9f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:23:30.088253 21792 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f95ded2ddf74ec88a694a3c96c6c9f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f95ded2ddf74ec88a694a3c96c6c9f1, State: Initialized, Role: FOLLOWER
I20260928 20:23:30.088366 21792 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f95ded2ddf74ec88a694a3c96c6c9f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f95ded2ddf74ec88a694a3c96c6c9f1" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 34549 } }
I20260928 20:23:30.088456 21792 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f95ded2ddf74ec88a694a3c96c6c9f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:23:30.088503 21792 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f95ded2ddf74ec88a694a3c96c6c9f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:23:30.088575 21792 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f95ded2ddf74ec88a694a3c96c6c9f1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:23:30.089231 21792 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f95ded2ddf74ec88a694a3c96c6c9f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f95ded2ddf74ec88a694a3c96c6c9f1" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 34549 } }
I20260928 20:23:30.089360 21792 leader_election.cc:302] T 00000000000000000000000000000000 P 2f95ded2ddf74ec88a694a3c96c6c9f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f95ded2ddf74ec88a694a3c96c6c9f1; no voters:
I20260928 20:23:30.089563 21792 leader_election.cc:288] T 00000000000000000000000000000000 P 2f95ded2ddf74ec88a694a3c96c6c9f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:23:30.089705 21795 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f95ded2ddf74ec88a694a3c96c6c9f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:23:30.089906 21792 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f95ded2ddf74ec88a694a3c96c6c9f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:23:30.089939 21795 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f95ded2ddf74ec88a694a3c96c6c9f1 [term 1 LEADER]: Becoming Leader. State: Replica: 2f95ded2ddf74ec88a694a3c96c6c9f1, State: Running, Role: LEADER
I20260928 20:23:30.090109 21795 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f95ded2ddf74ec88a694a3c96c6c9f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f95ded2ddf74ec88a694a3c96c6c9f1" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 34549 } }
I20260928 20:23:30.090504 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 21725
I20260928 20:23:30.090665 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790626764250411-3144-0/minicluster-data/master-0/wal/instance
I20260928 20:23:30.090644 21797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f95ded2ddf74ec88a694a3c96c6c9f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f95ded2ddf74ec88a694a3c96c6c9f1. Latest consensus state: current_term: 1 leader_uuid: "2f95ded2ddf74ec88a694a3c96c6c9f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f95ded2ddf74ec88a694a3c96c6c9f1" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 34549 } } }
I20260928 20:23:30.090729 21797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f95ded2ddf74ec88a694a3c96c6c9f1 [sys.catalog]: This master's current role is: LEADER
I20260928 20:23:30.090927 3144 external_mini_cluster.cc:1428] Setting key 617d111c84238f2ca8d710394b87b6d6
I20260928 20:23:30.090981 21804 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:23:30.091800 21804 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:23:30.091797 21796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f95ded2ddf74ec88a694a3c96c6c9f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f95ded2ddf74ec88a694a3c96c6c9f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f95ded2ddf74ec88a694a3c96c6c9f1" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 34549 } } }
I20260928 20:23:30.093122 21796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f95ded2ddf74ec88a694a3c96c6c9f1 [sys.catalog]: This master's current role is: LEADER
I20260928 20:23:30.094148 21804 catalog_manager.cc:1347] Generated new cluster ID: f3764b22c09e49b7a7d91fdf495b0e57
I20260928 20:23:30.094205 21804 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:23:30.097764 21813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:30.091603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57006 (local address 127.3.18.62:34549)
0928 20:23:30.091865 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:30.091877 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:30.092475 (+ 598us) server_negotiation.cc:407] Connection header received
0928 20:23:30.092662 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:30.092670 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:30.092760 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:30.092936 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:30.093211 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:30.094496 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:30.095440 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:30.096018 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:30.096200 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:30.097002 (+ 802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:30.097006 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:30.097006 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:23:30.097216 (+ 210us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:30.097252 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:23:30.097265 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:30.097353 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:30.097437 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 20:23:30.097536 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":92,"threads_started":1}
I20260928 20:23:30.118237 21804 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:23:30.118916 21804 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:23:30.121934 21804 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f95ded2ddf74ec88a694a3c96c6c9f1: Generated new TSK 0
I20260928 20:23:30.122135 21804 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 20:23:30.126747 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790626764250411-3144-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790626764250411-3144-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790626764250411-3144-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.1:0
--local_ip_for_outbound_sockets=127.3.18.1
--webserver_interface=127.3.18.1
--webserver_port=0
--tserver_master_addrs=127.3.18.62:34549
--builtin_ntp_servers=127.3.18.20:37567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 20:23:30.234035 21816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:23:30.234505 21816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:23:30.234609 21816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:23:30.236290 21816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:23:30.236367 21816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:23:30.236418 21816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.1
I20260928 20:23:30.238363 21816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:37567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790626764250411-3144-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790626764250411-3144-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.18.1
--webserver_port=0
--tserver_master_addrs=127.3.18.62:34549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.1
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790626764250411-3144-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:23:30.238708 21816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:23:30.239079 21816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:23:30.244776 21824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:30.244803 21821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:23:30.245064 21816 server_base.cc:1034] running on GCE node
W20260928 20:23:30.244776 21822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:23:30.245426 21816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:23:30.245754 21816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:23:30.246932 21816 hybrid_clock.cc:648] HybridClock initialized: now 1790627010246906 us; error 39 us; skew 500 ppm
I20260928 20:23:30.248479 21816 webserver.cc:460] Webserver started at http://127.3.18.1:37985/ using document root <none> and password file <none>
I20260928 20:23:30.248812 21816 fs_manager.cc:359] Metadata directory not provided
I20260928 20:23:30.248864 21816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:23:30.249001 21816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:23:30.250144 21816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790626764250411-3144-0/minicluster-data/ts-0/data/instance:
uuid: "f5d77357f81a44d893b3a48164de1a62"
format_stamp: "Formatted at 2026-09-28 20:23:30 on dist-test-slave-b83v"
server_key: "d66ab852a61237af670e1fb975fdd290"
server_key_iv: "fdc2b266d4eb65e493eb0e179364daab"
server_key_version: "encryptionkey@0"
I20260928 20:23:30.250516 21816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790626764250411-3144-0/minicluster-data/ts-0/wal/instance:
uuid: "f5d77357f81a44d893b3a48164de1a62"
format_stamp: "Formatted at 2026-09-28 20:23:30 on dist-test-slave-b83v"
server_key: "d66ab852a61237af670e1fb975fdd290"
server_key_iv: "fdc2b266d4eb65e493eb0e179364daab"
server_key_version: "encryptionkey@0"
I20260928 20:23:30.251986 21816 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260928 20:23:30.252825 21830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:30.253206 21816 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:23:30.253326 21816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790626764250411-3144-0/minicluster-data/ts-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790626764250411-3144-0/minicluster-data/ts-0/wal
uuid: "f5d77357f81a44d893b3a48164de1a62"
format_stamp: "Formatted at 2026-09-28 20:23:30 on dist-test-slave-b83v"
server_key: "d66ab852a61237af670e1fb975fdd290"
server_key_iv: "fdc2b266d4eb65e493eb0e179364daab"
server_key_version: "encryptionkey@0"
I20260928 20:23:30.253413 21816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790626764250411-3144-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790626764250411-3144-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790626764250411-3144-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:23:30.275741 21816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:23:30.276134 21816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:23:30.276290 21816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:23:30.276554 21816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:23:30.276924 21816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:23:30.276963 21816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:30.276986 21816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:23:30.276999 21816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:30.283409 21816 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.1:39625
I20260928 20:23:30.283478 21943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.1:39625 every 8 connection(s)
I20260928 20:23:30.283847 21816 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
I20260928 20:23:30.289258 21813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:30.284661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:59031 (local address 127.3.18.62:34549)
0928 20:23:30.284745 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:30.284748 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:30.285025 (+ 277us) server_negotiation.cc:407] Connection header received
0928 20:23:30.285465 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:30.285466 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:30.285480 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:30.285564 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:30.286247 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:30.286814 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:30.287859 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:30.288060 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:30.288118 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:30.288938 (+ 820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:30.288939 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:30.288939 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:23:30.288944 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:30.288958 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:23:30.288962 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:30.289010 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:30.289169 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 20:23:30.289194 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 20:23:30.289594 21946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:30.284644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:34549 (local address 127.3.18.1:59031)
0928 20:23:30.284827 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:30.284871 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 20:23:30.285361 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:23:30.285630 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:23:30.285634 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:23:30.285641 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:23:30.286153 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:30.286157 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:30.286912 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:30.286913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:30.287712 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:30.287714 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:30.288147 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:30.288147 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:30.288274 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:30.288801 (+ 527us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 20:23:30.288802 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:23:30.288831 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 20:23:30.288854 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:23:30.289059 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:23:30.289067 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:23:30.289071 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:23:30.289166 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 20:23:30.289286 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":81,"threads_started":1}
I20260928 20:23:30.290247 21944 heartbeater.cc:347] Connected to a master server at 127.3.18.62:34549
I20260928 20:23:30.290354 21944 heartbeater.cc:464] Registering TS with master...
I20260928 20:23:30.290560 21944 heartbeater.cc:511] Master 127.3.18.62:34549 requested a full tablet report, sending...
I20260928 20:23:30.291162 21756 ts_manager.cc:194] Registered new tserver with Master: f5d77357f81a44d893b3a48164de1a62 (127.3.18.1:39625)
I20260928 20:23:30.291934 21756 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.18.1:59031
I20260928 20:23:30.293628 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 21816
I20260928 20:23:30.293725 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790626764250411-3144-0/minicluster-data/ts-0/wal/instance
I20260928 20:23:30.293876 3144 external_mini_cluster.cc:1428] Setting key fc4092788c381d854d2435935fd7f8ba
I20260928 20:23:30.295226 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790626764250411-3144-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790626764250411-3144-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790626764250411-3144-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.2:0
--local_ip_for_outbound_sockets=127.3.18.2
--webserver_interface=127.3.18.2
--webserver_port=0
--tserver_master_addrs=127.3.18.62:34549
--builtin_ntp_servers=127.3.18.20:37567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 20:23:30.380208 21947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:23:30.380429 21947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:23:30.380453 21947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:23:30.381664 21947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:23:30.381714 21947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:23:30.381742 21947 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.2
I20260928 20:23:30.383128 21947 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:37567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790626764250411-3144-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790626764250411-3144-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.18.2
--webserver_port=0
--tserver_master_addrs=127.3.18.62:34549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.2
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790626764250411-3144-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:23:30.383349 21947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:23:30.383626 21947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:23:30.389123 21953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:30.389118 21952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:30.389118 21955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:23:30.389413 21947 server_base.cc:1034] running on GCE node
I20260928 20:23:30.389627 21947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:23:30.389855 21947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:23:30.390996 21947 hybrid_clock.cc:648] HybridClock initialized: now 1790627010390987 us; error 32 us; skew 500 ppm
I20260928 20:23:30.392445 21947 webserver.cc:460] Webserver started at http://127.3.18.2:42811/ using document root <none> and password file <none>
I20260928 20:23:30.392742 21947 fs_manager.cc:359] Metadata directory not provided
I20260928 20:23:30.392818 21947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:23:30.392963 21947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:23:30.394096 21947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790626764250411-3144-0/minicluster-data/ts-1/data/instance:
uuid: "1d442322d93d4a9b8f09e48df0c8dc36"
format_stamp: "Formatted at 2026-09-28 20:23:30 on dist-test-slave-b83v"
server_key: "c193b8c186b65221dfc596596047f2a0"
server_key_iv: "ecac116b1428676c317b1ea4c6760f0b"
server_key_version: "encryptionkey@0"
I20260928 20:23:30.394491 21947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790626764250411-3144-0/minicluster-data/ts-1/wal/instance:
uuid: "1d442322d93d4a9b8f09e48df0c8dc36"
format_stamp: "Formatted at 2026-09-28 20:23:30 on dist-test-slave-b83v"
server_key: "c193b8c186b65221dfc596596047f2a0"
server_key_iv: "ecac116b1428676c317b1ea4c6760f0b"
server_key_version: "encryptionkey@0"
I20260928 20:23:30.396065 21947 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:23:30.396920 21961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:30.397104 21947 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 20:23:30.397205 21947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790626764250411-3144-0/minicluster-data/ts-1/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790626764250411-3144-0/minicluster-data/ts-1/wal
uuid: "1d442322d93d4a9b8f09e48df0c8dc36"
format_stamp: "Formatted at 2026-09-28 20:23:30 on dist-test-slave-b83v"
server_key: "c193b8c186b65221dfc596596047f2a0"
server_key_iv: "ecac116b1428676c317b1ea4c6760f0b"
server_key_version: "encryptionkey@0"
I20260928 20:23:30.397300 21947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790626764250411-3144-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790626764250411-3144-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790626764250411-3144-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:23:30.408334 21947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:23:30.408717 21947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:23:30.408882 21947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:23:30.409137 21947 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:23:30.409482 21947 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:23:30.409535 21947 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:30.409608 21947 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:23:30.409638 21947 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:30.415473 21947 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.2:46307
I20260928 20:23:30.415560 22074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.2:46307 every 8 connection(s)
I20260928 20:23:30.415994 21947 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
I20260928 20:23:30.420817 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 21947
I20260928 20:23:30.420940 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790626764250411-3144-0/minicluster-data/ts-1/wal/instance
I20260928 20:23:30.421162 3144 external_mini_cluster.cc:1428] Setting key ebb992ebac9c780bf5efbc734a6dd88a
I20260928 20:23:30.421831 21813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:30.416803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:53669 (local address 127.3.18.62:34549)
0928 20:23:30.416870 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:30.416872 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:30.417088 (+ 216us) server_negotiation.cc:407] Connection header received
0928 20:23:30.417606 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:30.417608 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:30.417622 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:30.417707 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:30.418411 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:30.418953 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:30.420138 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:30.420349 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:30.420432 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:30.421422 (+ 990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:30.421424 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:30.421424 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:23:30.421430 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:30.421449 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:23:30.421454 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:30.421514 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:30.421694 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 20:23:30.421737 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 20:23:30.421988 22077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:30.416758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:34549 (local address 127.3.18.2:53669)
0928 20:23:30.416950 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:30.416992 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 20:23:30.417485 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:23:30.417773 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:23:30.417777 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:23:30.417785 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:23:30.418302 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:30.418306 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:30.419060 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:30.419062 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:30.420025 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:30.420028 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:30.420438 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:30.420438 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:30.420549 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:30.421246 (+ 697us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 20:23:30.421247 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:23:30.421290 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 20:23:30.421320 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:23:30.421536 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:23:30.421542 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:23:30.421546 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:23:30.421647 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 20:23:30.421736 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":98,"threads_started":1}
I20260928 20:23:30.422302 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790626764250411-3144-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790626764250411-3144-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790626764250411-3144-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.3:0
--local_ip_for_outbound_sockets=127.3.18.3
--webserver_interface=127.3.18.3
--webserver_port=0
--tserver_master_addrs=127.3.18.62:34549
--builtin_ntp_servers=127.3.18.20:37567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 20:23:30.422459 22075 heartbeater.cc:347] Connected to a master server at 127.3.18.62:34549
I20260928 20:23:30.422595 22075 heartbeater.cc:464] Registering TS with master...
I20260928 20:23:30.422791 22075 heartbeater.cc:511] Master 127.3.18.62:34549 requested a full tablet report, sending...
I20260928 20:23:30.423184 21756 ts_manager.cc:194] Registered new tserver with Master: 1d442322d93d4a9b8f09e48df0c8dc36 (127.3.18.2:46307)
I20260928 20:23:30.423715 21756 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.18.2:53669
W20260928 20:23:30.511266 22078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:23:30.511576 22078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:23:30.511622 22078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:23:30.512974 22078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:23:30.513127 22078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:23:30.513181 22078 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.3
I20260928 20:23:30.514649 22078 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:37567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790626764250411-3144-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790626764250411-3144-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.18.3
--webserver_port=0
--tserver_master_addrs=127.3.18.62:34549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.3
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790626764250411-3144-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:23:30.514978 22078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:23:30.515309 22078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 20:23:30.520641 22078 server_base.cc:1034] running on GCE node
W20260928 20:23:30.520687 22084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:30.520787 22086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:30.520787 22083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:23:30.521062 22078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:23:30.521284 22078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:23:30.522481 22078 hybrid_clock.cc:648] HybridClock initialized: now 1790627010522465 us; error 27 us; skew 500 ppm
I20260928 20:23:30.524113 22078 webserver.cc:460] Webserver started at http://127.3.18.3:38889/ using document root <none> and password file <none>
I20260928 20:23:30.524427 22078 fs_manager.cc:359] Metadata directory not provided
I20260928 20:23:30.524518 22078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:23:30.524705 22078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:23:30.525980 22078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790626764250411-3144-0/minicluster-data/ts-2/data/instance:
uuid: "a2001971f10b476d9c4ebe57d2989232"
format_stamp: "Formatted at 2026-09-28 20:23:30 on dist-test-slave-b83v"
server_key: "06f0b1cb56824ead4101f843b49c9d5b"
server_key_iv: "25fec0b522bf60dee7f123a1f50deaa0"
server_key_version: "encryptionkey@0"
I20260928 20:23:30.526408 22078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790626764250411-3144-0/minicluster-data/ts-2/wal/instance:
uuid: "a2001971f10b476d9c4ebe57d2989232"
format_stamp: "Formatted at 2026-09-28 20:23:30 on dist-test-slave-b83v"
server_key: "06f0b1cb56824ead4101f843b49c9d5b"
server_key_iv: "25fec0b522bf60dee7f123a1f50deaa0"
server_key_version: "encryptionkey@0"
I20260928 20:23:30.528120 22078 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260928 20:23:30.529091 22092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:30.529305 22078 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:23:30.529429 22078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790626764250411-3144-0/minicluster-data/ts-2/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790626764250411-3144-0/minicluster-data/ts-2/wal
uuid: "a2001971f10b476d9c4ebe57d2989232"
format_stamp: "Formatted at 2026-09-28 20:23:30 on dist-test-slave-b83v"
server_key: "06f0b1cb56824ead4101f843b49c9d5b"
server_key_iv: "25fec0b522bf60dee7f123a1f50deaa0"
server_key_version: "encryptionkey@0"
I20260928 20:23:30.529513 22078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790626764250411-3144-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790626764250411-3144-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790626764250411-3144-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:23:30.547410 22078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:23:30.547916 22078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:23:30.548091 22078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:23:30.548364 22078 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:23:30.548722 22078 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:23:30.548779 22078 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:30.548820 22078 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:23:30.548853 22078 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:30.554616 22078 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.3:45875
I20260928 20:23:30.554775 22205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.3:45875 every 8 connection(s)
I20260928 20:23:30.555068 22078 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
I20260928 20:23:30.558290 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 22078
I20260928 20:23:30.558383 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790626764250411-3144-0/minicluster-data/ts-2/wal/instance
I20260928 20:23:30.558593 3144 external_mini_cluster.cc:1428] Setting key 2cda9be17ca864876b2bd2699eb6b771
I20260928 20:23:30.561333 21813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:30.556014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:41587 (local address 127.3.18.62:34549)
0928 20:23:30.556058 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:30.556061 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:30.556369 (+ 308us) server_negotiation.cc:407] Connection header received
0928 20:23:30.557030 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:30.557032 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:30.557045 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:30.557115 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:30.557829 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:30.558381 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:30.559797 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:30.560019 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:30.560102 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:30.561039 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:30.561041 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:30.561041 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:23:30.561047 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:30.561061 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:23:30.561065 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:30.561113 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:30.561211 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 20:23:30.561247 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 20:23:30.561559 22208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:30.556030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:34549 (local address 127.3.18.3:41587)
0928 20:23:30.556238 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:30.556273 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 20:23:30.556914 (+ 641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:23:30.557197 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:23:30.557200 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:23:30.557209 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:23:30.557719 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:30.557723 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:30.558478 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:30.558480 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:30.559670 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:30.559673 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:30.560157 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:30.560158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:30.560293 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:30.560864 (+ 571us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 20:23:30.560865 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:23:30.560897 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 20:23:30.560929 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:23:30.561115 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:23:30.561120 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:23:30.561124 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:23:30.561213 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 20:23:30.561319 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":105,"threads_started":1}
I20260928 20:23:30.562193 22206 heartbeater.cc:347] Connected to a master server at 127.3.18.62:34549
I20260928 20:23:30.562286 22206 heartbeater.cc:464] Registering TS with master...
I20260928 20:23:30.562438 22206 heartbeater.cc:511] Master 127.3.18.62:34549 requested a full tablet report, sending...
I20260928 20:23:30.562801 21756 ts_manager.cc:194] Registered new tserver with Master: a2001971f10b476d9c4ebe57d2989232 (127.3.18.3:45875)
I20260928 20:23:30.563182 21756 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.18.3:41587
I20260928 20:23:30.570492 3144 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 20:23:30.575314 21813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:30.571983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57018 (local address 127.3.18.62:34549)
0928 20:23:30.572051 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:30.572055 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:30.572180 (+ 125us) server_negotiation.cc:407] Connection header received
0928 20:23:30.572253 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:30.572254 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:30.572267 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:30.572338 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:30.572521 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:30.573029 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:30.573871 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:30.574079 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:30.574185 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:30.574952 (+ 767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:30.574953 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:30.574954 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:23:30.574959 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:30.574980 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:23:30.574984 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:30.575084 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:30.575208 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 20:23:30.575221 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 20:23:30.577171 21756 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57018:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 20:23:30.578769 21756 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 20:23:30.585863 22214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:30.581279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.1:39625 (local address 127.0.0.1:47216)
0928 20:23:30.581417 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:30.581459 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 20:23:30.581523 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:23:30.581898 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:23:30.581901 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:23:30.581902 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:23:30.581998 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:30.582000 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:30.583358 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:30.583359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:30.584536 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:30.584539 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:30.585475 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:30.585476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:30.585608 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:30.585628 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 20:23:30.585709 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 20:23:30.585769 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":36,"thread_start_us":89,"threads_started":1}
I20260928 20:23:30.585875 22215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:30.581371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47216 (local address 127.3.18.1:39625)
0928 20:23:30.581575 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:30.581582 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:30.581598 (+ 16us) server_negotiation.cc:407] Connection header received
0928 20:23:30.581694 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:30.581697 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:30.581714 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:30.581809 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:23:30.582092 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:30.583263 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:30.584629 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:30.585388 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:30.585469 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:30.585567 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:30.585742 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 20:23:30.585785 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":135,"threads_started":1}
I20260928 20:23:30.586349 22216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:30.581656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.2:46307 (local address 127.0.0.1:57378)
0928 20:23:30.582089 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:30.582099 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 20:23:30.582146 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:23:30.582431 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:23:30.582432 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:23:30.582433 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:23:30.582479 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:30.582480 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:30.583354 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:30.583355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:30.584750 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:30.584753 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:30.585628 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:30.585629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:30.585689 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:30.585693 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:23:30.585745 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 20:23:30.585774 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"mutex_wait_us":57,"thread_start_us":69,"threads_started":1}
I20260928 20:23:30.586515 22218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:30.581777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57378 (local address 127.3.18.2:46307)
0928 20:23:30.582273 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:30.582277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:30.582286 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:23:30.582321 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:30.582323 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:30.582337 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:30.582433 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:23:30.582534 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:30.583264 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:30.584842 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:30.585545 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:30.585622 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:30.585709 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:30.585814 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 20:23:30.585859 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":77,"threads_started":1}
I20260928 20:23:30.586782 21878 tablet_service.cc:1467] Processing CreateTablet for tablet 1f77dfbfc5114adaa729cf45f7698099 (DEFAULT_TABLE table=test-table [id=e2f44fc3473f4ed98f825af2a6957f8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:23:30.586973 22217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:30.581510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.3:45875 (local address 127.0.0.1:38138)
0928 20:23:30.581791 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:30.581802 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:23:30.581847 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:23:30.582252 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:23:30.582253 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:23:30.582255 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:23:30.582292 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:30.582294 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:30.583541 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:30.583542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:30.584750 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:30.584753 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:30.586734 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:30.586735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:30.586797 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:30.586801 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:23:30.586857 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 20:23:30.586901 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":28,"thread_start_us":48,"threads_started":1}
I20260928 20:23:30.586983 22219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:30.581837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38138 (local address 127.3.18.3:45875)
0928 20:23:30.582015 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:30.582019 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:30.582029 (+ 10us) server_negotiation.cc:407] Connection header received
0928 20:23:30.582068 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:30.582070 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:30.582087 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:30.582362 (+ 275us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:23:30.582428 (+ 66us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:30.583459 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:30.585758 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:30.586634 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:30.586730 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:30.586821 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:30.586870 (+ 49us) server_negotiation.cc:299] Negotiation successful
0928 20:23:30.586907 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":94,"threads_started":1}
I20260928 20:23:30.587194 21878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f77dfbfc5114adaa729cf45f7698099. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:23:30.587625 22140 tablet_service.cc:1467] Processing CreateTablet for tablet 1f77dfbfc5114adaa729cf45f7698099 (DEFAULT_TABLE table=test-table [id=e2f44fc3473f4ed98f825af2a6957f8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:23:30.588007 22140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f77dfbfc5114adaa729cf45f7698099. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:23:30.588238 22009 tablet_service.cc:1467] Processing CreateTablet for tablet 1f77dfbfc5114adaa729cf45f7698099 (DEFAULT_TABLE table=test-table [id=e2f44fc3473f4ed98f825af2a6957f8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:23:30.588564 22009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f77dfbfc5114adaa729cf45f7698099. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:23:30.590165 22220 tablet_bootstrap.cc:492] T 1f77dfbfc5114adaa729cf45f7698099 P f5d77357f81a44d893b3a48164de1a62: Bootstrap starting.
I20260928 20:23:30.590946 22220 tablet_bootstrap.cc:654] T 1f77dfbfc5114adaa729cf45f7698099 P f5d77357f81a44d893b3a48164de1a62: Neither blocks nor log segments found. Creating new log.
I20260928 20:23:30.591346 22220 log.cc:824] T 1f77dfbfc5114adaa729cf45f7698099 P f5d77357f81a44d893b3a48164de1a62: Log is configured to *not* fsync() on all Append() calls
I20260928 20:23:30.591421 22221 tablet_bootstrap.cc:492] T 1f77dfbfc5114adaa729cf45f7698099 P a2001971f10b476d9c4ebe57d2989232: Bootstrap starting.
I20260928 20:23:30.592212 22220 tablet_bootstrap.cc:492] T 1f77dfbfc5114adaa729cf45f7698099 P f5d77357f81a44d893b3a48164de1a62: No bootstrap required, opened a new log
I20260928 20:23:30.592257 22221 tablet_bootstrap.cc:654] T 1f77dfbfc5114adaa729cf45f7698099 P a2001971f10b476d9c4ebe57d2989232: Neither blocks nor log segments found. Creating new log.
I20260928 20:23:30.592294 22220 ts_tablet_manager.cc:1397] T 1f77dfbfc5114adaa729cf45f7698099 P f5d77357f81a44d893b3a48164de1a62: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 20:23:30.592474 22223 tablet_bootstrap.cc:492] T 1f77dfbfc5114adaa729cf45f7698099 P 1d442322d93d4a9b8f09e48df0c8dc36: Bootstrap starting.
I20260928 20:23:30.592585 22221 log.cc:824] T 1f77dfbfc5114adaa729cf45f7698099 P a2001971f10b476d9c4ebe57d2989232: Log is configured to *not* fsync() on all Append() calls
I20260928 20:23:30.593130 22223 tablet_bootstrap.cc:654] T 1f77dfbfc5114adaa729cf45f7698099 P 1d442322d93d4a9b8f09e48df0c8dc36: Neither blocks nor log segments found. Creating new log.
I20260928 20:23:30.593392 22221 tablet_bootstrap.cc:492] T 1f77dfbfc5114adaa729cf45f7698099 P a2001971f10b476d9c4ebe57d2989232: No bootstrap required, opened a new log
I20260928 20:23:30.593425 22223 log.cc:824] T 1f77dfbfc5114adaa729cf45f7698099 P 1d442322d93d4a9b8f09e48df0c8dc36: Log is configured to *not* fsync() on all Append() calls
I20260928 20:23:30.593482 22221 ts_tablet_manager.cc:1397] T 1f77dfbfc5114adaa729cf45f7698099 P a2001971f10b476d9c4ebe57d2989232: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 20:23:30.594091 22223 tablet_bootstrap.cc:492] T 1f77dfbfc5114adaa729cf45f7698099 P 1d442322d93d4a9b8f09e48df0c8dc36: No bootstrap required, opened a new log
I20260928 20:23:30.594219 22223 ts_tablet_manager.cc:1397] T 1f77dfbfc5114adaa729cf45f7698099 P 1d442322d93d4a9b8f09e48df0c8dc36: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 20:23:30.595065 22220 raft_consensus.cc:348] T 1f77dfbfc5114adaa729cf45f7698099 P f5d77357f81a44d893b3a48164de1a62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d77357f81a44d893b3a48164de1a62" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 39625 } } peers { permanent_uuid: "a2001971f10b476d9c4ebe57d2989232" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 45875 } } peers { permanent_uuid: "1d442322d93d4a9b8f09e48df0c8dc36" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 46307 } }
I20260928 20:23:30.595242 22220 raft_consensus.cc:374] T 1f77dfbfc5114adaa729cf45f7698099 P f5d77357f81a44d893b3a48164de1a62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:23:30.595278 22220 raft_consensus.cc:729] T 1f77dfbfc5114adaa729cf45f7698099 P f5d77357f81a44d893b3a48164de1a62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5d77357f81a44d893b3a48164de1a62, State: Initialized, Role: FOLLOWER
I20260928 20:23:30.595444 22220 consensus_queue.cc:260] T 1f77dfbfc5114adaa729cf45f7698099 P f5d77357f81a44d893b3a48164de1a62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d77357f81a44d893b3a48164de1a62" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 39625 } } peers { permanent_uuid: "a2001971f10b476d9c4ebe57d2989232" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 45875 } } peers { permanent_uuid: "1d442322d93d4a9b8f09e48df0c8dc36" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 46307 } }
I20260928 20:23:30.595717 22220 ts_tablet_manager.cc:1428] T 1f77dfbfc5114adaa729cf45f7698099 P f5d77357f81a44d893b3a48164de1a62: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 20:23:30.595881 21944 heartbeater.cc:503] Master 127.3.18.62:34549 was elected leader, sending a full tablet report...
I20260928 20:23:30.596418 22221 raft_consensus.cc:348] T 1f77dfbfc5114adaa729cf45f7698099 P a2001971f10b476d9c4ebe57d2989232 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d77357f81a44d893b3a48164de1a62" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 39625 } } peers { permanent_uuid: "a2001971f10b476d9c4ebe57d2989232" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 45875 } } peers { permanent_uuid: "1d442322d93d4a9b8f09e48df0c8dc36" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 46307 } }
I20260928 20:23:30.596575 22221 raft_consensus.cc:374] T 1f77dfbfc5114adaa729cf45f7698099 P a2001971f10b476d9c4ebe57d2989232 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:23:30.596619 22221 raft_consensus.cc:729] T 1f77dfbfc5114adaa729cf45f7698099 P a2001971f10b476d9c4ebe57d2989232 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2001971f10b476d9c4ebe57d2989232, State: Initialized, Role: FOLLOWER
I20260928 20:23:30.596771 22223 raft_consensus.cc:348] T 1f77dfbfc5114adaa729cf45f7698099 P 1d442322d93d4a9b8f09e48df0c8dc36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d77357f81a44d893b3a48164de1a62" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 39625 } } peers { permanent_uuid: "a2001971f10b476d9c4ebe57d2989232" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 45875 } } peers { permanent_uuid: "1d442322d93d4a9b8f09e48df0c8dc36" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 46307 } }
I20260928 20:23:30.596884 22221 consensus_queue.cc:260] T 1f77dfbfc5114adaa729cf45f7698099 P a2001971f10b476d9c4ebe57d2989232 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d77357f81a44d893b3a48164de1a62" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 39625 } } peers { permanent_uuid: "a2001971f10b476d9c4ebe57d2989232" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 45875 } } peers { permanent_uuid: "1d442322d93d4a9b8f09e48df0c8dc36" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 46307 } }
I20260928 20:23:30.596953 22223 raft_consensus.cc:374] T 1f77dfbfc5114adaa729cf45f7698099 P 1d442322d93d4a9b8f09e48df0c8dc36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:23:30.596992 22223 raft_consensus.cc:729] T 1f77dfbfc5114adaa729cf45f7698099 P 1d442322d93d4a9b8f09e48df0c8dc36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d442322d93d4a9b8f09e48df0c8dc36, State: Initialized, Role: FOLLOWER
I20260928 20:23:30.597107 22223 consensus_queue.cc:260] T 1f77dfbfc5114adaa729cf45f7698099 P 1d442322d93d4a9b8f09e48df0c8dc36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d77357f81a44d893b3a48164de1a62" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 39625 } } peers { permanent_uuid: "a2001971f10b476d9c4ebe57d2989232" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 45875 } } peers { permanent_uuid: "1d442322d93d4a9b8f09e48df0c8dc36" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 46307 } }
I20260928 20:23:30.597173 22221 ts_tablet_manager.cc:1428] T 1f77dfbfc5114adaa729cf45f7698099 P a2001971f10b476d9c4ebe57d2989232: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260928 20:23:30.597319 22206 heartbeater.cc:503] Master 127.3.18.62:34549 was elected leader, sending a full tablet report...
I20260928 20:23:30.597364 22223 ts_tablet_manager.cc:1428] T 1f77dfbfc5114adaa729cf45f7698099 P 1d442322d93d4a9b8f09e48df0c8dc36: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 20:23:30.597478 22075 heartbeater.cc:503] Master 127.3.18.62:34549 was elected leader, sending a full tablet report...
W20260928 20:23:30.666574 22076 tablet.cc:2369] T 1f77dfbfc5114adaa729cf45f7698099 P 1d442322d93d4a9b8f09e48df0c8dc36: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 20:23:30.667723 22228 raft_consensus.cc:482] T 1f77dfbfc5114adaa729cf45f7698099 P 1d442322d93d4a9b8f09e48df0c8dc36 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:23:30.667878 22228 raft_consensus.cc:504] T 1f77dfbfc5114adaa729cf45f7698099 P 1d442322d93d4a9b8f09e48df0c8dc36 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d77357f81a44d893b3a48164de1a62" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 39625 } } peers { permanent_uuid: "a2001971f10b476d9c4ebe57d2989232" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 45875 } } peers { permanent_uuid: "1d442322d93d4a9b8f09e48df0c8dc36" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 46307 } }
I20260928 20:23:30.668231 22228 leader_election.cc:288] T 1f77dfbfc5114adaa729cf45f7698099 P 1d442322d93d4a9b8f09e48df0c8dc36 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f5d77357f81a44d893b3a48164de1a62 (127.3.18.1:39625), a2001971f10b476d9c4ebe57d2989232 (127.3.18.3:45875)
I20260928 20:23:30.671286 22077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:30.668425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.1:39625 (local address 127.3.18.2:38531)
0928 20:23:30.668496 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:30.668512 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 20:23:30.668558 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:23:30.668837 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:23:30.668838 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:23:30.668840 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:23:30.668916 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:30.668918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:30.669651 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:30.669652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:30.670495 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:30.670498 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:30.671014 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:30.671015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:30.671088 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:30.671093 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 20:23:30.671149 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 20:23:30.671200 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":46}
I20260928 20:23:30.671310 22215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:30.668533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:38531 (local address 127.3.18.1:39625)
0928 20:23:30.668570 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:30.668573 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:30.668705 (+ 132us) server_negotiation.cc:407] Connection header received
0928 20:23:30.668739 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:30.668740 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:30.668756 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:30.668817 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:23:30.668999 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:30.669513 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:30.670585 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:30.670943 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:30.670998 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:30.671006 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:30.671191 (+ 185us) server_negotiation.cc:299] Negotiation successful
0928 20:23:30.671234 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 20:23:30.671499 22229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:30.668575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.3:45875 (local address 127.3.18.2:55015)
0928 20:23:30.668739 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:30.668749 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 20:23:30.668801 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:23:30.668917 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:23:30.668918 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:23:30.668919 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:23:30.669006 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:30.669007 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:30.669608 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:30.669609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:30.670419 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:30.670421 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:30.671210 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:30.671211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:30.671287 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:30.671291 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:23:30.671354 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 20:23:30.671398 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":104,"threads_started":1}
I20260928 20:23:30.671528 22219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:30.668730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:55015 (local address 127.3.18.3:45875)
0928 20:23:30.668772 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:30.668774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:30.668813 (+ 39us) server_negotiation.cc:407] Connection header received
0928 20:23:30.668845 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:30.668846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:30.668858 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:30.668917 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:23:30.669076 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:30.669518 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:30.670505 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:30.671064 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:30.671151 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:30.671161 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:30.671373 (+ 212us) server_negotiation.cc:299] Negotiation successful
0928 20:23:30.671431 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 20:23:30.671967 21898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f77dfbfc5114adaa729cf45f7698099" candidate_uuid: "1d442322d93d4a9b8f09e48df0c8dc36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5d77357f81a44d893b3a48164de1a62" is_pre_election: true
I20260928 20:23:30.671977 22160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f77dfbfc5114adaa729cf45f7698099" candidate_uuid: "1d442322d93d4a9b8f09e48df0c8dc36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2001971f10b476d9c4ebe57d2989232" is_pre_election: true
I20260928 20:23:30.672135 21898 raft_consensus.cc:2454] T 1f77dfbfc5114adaa729cf45f7698099 P f5d77357f81a44d893b3a48164de1a62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d442322d93d4a9b8f09e48df0c8dc36 in term 0.
I20260928 20:23:30.672140 22160 raft_consensus.cc:2454] T 1f77dfbfc5114adaa729cf45f7698099 P a2001971f10b476d9c4ebe57d2989232 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d442322d93d4a9b8f09e48df0c8dc36 in term 0.
I20260928 20:23:30.672405 21963 leader_election.cc:302] T 1f77dfbfc5114adaa729cf45f7698099 P 1d442322d93d4a9b8f09e48df0c8dc36 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d442322d93d4a9b8f09e48df0c8dc36, f5d77357f81a44d893b3a48164de1a62; no voters:
I20260928 20:23:30.672533 22228 raft_consensus.cc:2781] T 1f77dfbfc5114adaa729cf45f7698099 P 1d442322d93d4a9b8f09e48df0c8dc36 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 20:23:30.672587 22228 raft_consensus.cc:482] T 1f77dfbfc5114adaa729cf45f7698099 P 1d442322d93d4a9b8f09e48df0c8dc36 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 20:23:30.672621 22228 raft_consensus.cc:3037] T 1f77dfbfc5114adaa729cf45f7698099 P 1d442322d93d4a9b8f09e48df0c8dc36 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:23:30.673332 22228 raft_consensus.cc:504] T 1f77dfbfc5114adaa729cf45f7698099 P 1d442322d93d4a9b8f09e48df0c8dc36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d77357f81a44d893b3a48164de1a62" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 39625 } } peers { permanent_uuid: "a2001971f10b476d9c4ebe57d2989232" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 45875 } } peers { permanent_uuid: "1d442322d93d4a9b8f09e48df0c8dc36" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 46307 } }
I20260928 20:23:30.673470 22228 leader_election.cc:288] T 1f77dfbfc5114adaa729cf45f7698099 P 1d442322d93d4a9b8f09e48df0c8dc36 [CANDIDATE]: Term 1 election: Requested vote from peers f5d77357f81a44d893b3a48164de1a62 (127.3.18.1:39625), a2001971f10b476d9c4ebe57d2989232 (127.3.18.3:45875)
I20260928 20:23:30.673681 22160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f77dfbfc5114adaa729cf45f7698099" candidate_uuid: "1d442322d93d4a9b8f09e48df0c8dc36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2001971f10b476d9c4ebe57d2989232"
I20260928 20:23:30.673704 21898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f77dfbfc5114adaa729cf45f7698099" candidate_uuid: "1d442322d93d4a9b8f09e48df0c8dc36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5d77357f81a44d893b3a48164de1a62"
I20260928 20:23:30.673767 22160 raft_consensus.cc:3037] T 1f77dfbfc5114adaa729cf45f7698099 P a2001971f10b476d9c4ebe57d2989232 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:23:30.673790 21898 raft_consensus.cc:3037] T 1f77dfbfc5114adaa729cf45f7698099 P f5d77357f81a44d893b3a48164de1a62 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:23:30.674500 22160 raft_consensus.cc:2454] T 1f77dfbfc5114adaa729cf45f7698099 P a2001971f10b476d9c4ebe57d2989232 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d442322d93d4a9b8f09e48df0c8dc36 in term 1.
I20260928 20:23:30.674650 21898 raft_consensus.cc:2454] T 1f77dfbfc5114adaa729cf45f7698099 P f5d77357f81a44d893b3a48164de1a62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d442322d93d4a9b8f09e48df0c8dc36 in term 1.
I20260928 20:23:30.674672 21963 leader_election.cc:302] T 1f77dfbfc5114adaa729cf45f7698099 P 1d442322d93d4a9b8f09e48df0c8dc36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d442322d93d4a9b8f09e48df0c8dc36, a2001971f10b476d9c4ebe57d2989232; no voters:
I20260928 20:23:30.674752 22228 raft_consensus.cc:2781] T 1f77dfbfc5114adaa729cf45f7698099 P 1d442322d93d4a9b8f09e48df0c8dc36 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:23:30.674927 22228 raft_consensus.cc:686] T 1f77dfbfc5114adaa729cf45f7698099 P 1d442322d93d4a9b8f09e48df0c8dc36 [term 1 LEADER]: Becoming Leader. State: Replica: 1d442322d93d4a9b8f09e48df0c8dc36, State: Running, Role: LEADER
I20260928 20:23:30.675064 22228 consensus_queue.cc:237] T 1f77dfbfc5114adaa729cf45f7698099 P 1d442322d93d4a9b8f09e48df0c8dc36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d77357f81a44d893b3a48164de1a62" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 39625 } } peers { permanent_uuid: "a2001971f10b476d9c4ebe57d2989232" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 45875 } } peers { permanent_uuid: "1d442322d93d4a9b8f09e48df0c8dc36" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 46307 } }
I20260928 20:23:30.675952 21755 catalog_manager.cc:5495] T 1f77dfbfc5114adaa729cf45f7698099 P 1d442322d93d4a9b8f09e48df0c8dc36 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d442322d93d4a9b8f09e48df0c8dc36 (127.3.18.2). New cstate: current_term: 1 leader_uuid: "1d442322d93d4a9b8f09e48df0c8dc36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d77357f81a44d893b3a48164de1a62" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 39625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2001971f10b476d9c4ebe57d2989232" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 45875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d442322d93d4a9b8f09e48df0c8dc36" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 46307 } health_report { overall_health: HEALTHY } } }
I20260928 20:23:30.685248 22218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:30.682328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57392 (local address 127.3.18.2:46307)
0928 20:23:30.682395 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:30.682397 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:30.682406 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:23:30.682448 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:30.682449 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:30.682480 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:30.682572 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:23:30.682794 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:30.683331 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:30.684318 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:30.684556 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:30.684652 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:30.684824 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:23:30.684936 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:23:30.685063 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:30.685106 (+ 43us) server_negotiation.cc:299] Negotiation successful
0928 20:23:30.685119 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 20:23:30.687445 22160 raft_consensus.cc:1261] T 1f77dfbfc5114adaa729cf45f7698099 P a2001971f10b476d9c4ebe57d2989232 [term 1 FOLLOWER]: Refusing update from remote peer 1d442322d93d4a9b8f09e48df0c8dc36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:23:30.687443 21898 raft_consensus.cc:1261] T 1f77dfbfc5114adaa729cf45f7698099 P f5d77357f81a44d893b3a48164de1a62 [term 1 FOLLOWER]: Refusing update from remote peer 1d442322d93d4a9b8f09e48df0c8dc36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:23:30.687920 22230 consensus_queue.cc:1035] T 1f77dfbfc5114adaa729cf45f7698099 P 1d442322d93d4a9b8f09e48df0c8dc36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2001971f10b476d9c4ebe57d2989232" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 45875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:23:30.688094 22228 consensus_queue.cc:1035] T 1f77dfbfc5114adaa729cf45f7698099 P 1d442322d93d4a9b8f09e48df0c8dc36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5d77357f81a44d893b3a48164de1a62" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 39625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:23:30.690812 22233 mvcc.cc:205] Tried to move back new op lower bound from 7334408235772706816 to 7334408235725877248. Current Snapshot: MvccSnapshot[applied={T|T < 7334408235772706816}]
I20260928 20:23:30.691506 22235 mvcc.cc:205] Tried to move back new op lower bound from 7334408235772706816 to 7334408235725877248. Current Snapshot: MvccSnapshot[applied={T|T < 7334408235772706816}]
I20260928 20:23:30.693920 21755 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57018:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:23:30.694037 21755 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57018:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:23:30.694960 21755 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2f95ded2ddf74ec88a694a3c96c6c9f1: Sending DeleteTablet for 3 replicas of tablet 1f77dfbfc5114adaa729cf45f7698099
I20260928 20:23:30.695420 21878 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f77dfbfc5114adaa729cf45f7698099 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:23:30 UTC) from {username='slave'} at 127.0.0.1:47216
I20260928 20:23:30.695484 22140 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f77dfbfc5114adaa729cf45f7698099 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:23:30 UTC) from {username='slave'} at 127.0.0.1:38138
I20260928 20:23:30.695658 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 21816
I20260928 20:23:30.695695 22243 tablet_replica.cc:330] T 1f77dfbfc5114adaa729cf45f7698099 P f5d77357f81a44d893b3a48164de1a62: stopping tablet replica
I20260928 20:23:30.698411 22009 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f77dfbfc5114adaa729cf45f7698099 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:23:30 UTC) from {username='slave'} at 127.0.0.1:57378
I20260928 20:23:30.698757 22245 tablet_replica.cc:330] T 1f77dfbfc5114adaa729cf45f7698099 P 1d442322d93d4a9b8f09e48df0c8dc36: stopping tablet replica
I20260928 20:23:30.698876 22245 raft_consensus.cc:2229] T 1f77dfbfc5114adaa729cf45f7698099 P 1d442322d93d4a9b8f09e48df0c8dc36 [term 1 LEADER]: Raft consensus shutting down.
I20260928 20:23:30.699015 22245 raft_consensus.cc:2258] T 1f77dfbfc5114adaa729cf45f7698099 P 1d442322d93d4a9b8f09e48df0c8dc36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:23:30.699527 22245 ts_tablet_manager.cc:1905] T 1f77dfbfc5114adaa729cf45f7698099 P 1d442322d93d4a9b8f09e48df0c8dc36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:23:30.699899 22244 tablet_replica.cc:330] T 1f77dfbfc5114adaa729cf45f7698099 P a2001971f10b476d9c4ebe57d2989232: stopping tablet replica
I20260928 20:23:30.700100 22244 raft_consensus.cc:2229] T 1f77dfbfc5114adaa729cf45f7698099 P a2001971f10b476d9c4ebe57d2989232 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:23:30.700300 22244 raft_consensus.cc:2258] T 1f77dfbfc5114adaa729cf45f7698099 P a2001971f10b476d9c4ebe57d2989232 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:23:30.700932 22244 ts_tablet_manager.cc:1905] T 1f77dfbfc5114adaa729cf45f7698099 P a2001971f10b476d9c4ebe57d2989232: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:23:30.701571 22245 ts_tablet_manager.cc:1918] T 1f77dfbfc5114adaa729cf45f7698099 P 1d442322d93d4a9b8f09e48df0c8dc36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:23:30.701663 22245 log.cc:1196] T 1f77dfbfc5114adaa729cf45f7698099 P 1d442322d93d4a9b8f09e48df0c8dc36: Deleting WAL directory at /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790626764250411-3144-0/minicluster-data/ts-1/wal/wals/1f77dfbfc5114adaa729cf45f7698099
I20260928 20:23:30.701952 22245 ts_tablet_manager.cc:1939] T 1f77dfbfc5114adaa729cf45f7698099 P 1d442322d93d4a9b8f09e48df0c8dc36: Deleting consensus metadata
I20260928 20:23:30.702411 21741 catalog_manager.cc:4841] TS 1d442322d93d4a9b8f09e48df0c8dc36 (127.3.18.2:46307): tablet 1f77dfbfc5114adaa729cf45f7698099 (table test-table [id=e2f44fc3473f4ed98f825af2a6957f8a]) successfully deleted
W20260928 20:23:30.703300 21741 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.18.1:39625 (error 108)
I20260928 20:23:30.703531 22244 ts_tablet_manager.cc:1918] T 1f77dfbfc5114adaa729cf45f7698099 P a2001971f10b476d9c4ebe57d2989232: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:23:30.703601 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 21947
I20260928 20:23:30.703852 22217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:30.703701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.1:39625 (local address 127.0.0.1:47228)
0928 20:23:30.703747 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:30.703812 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.1:39625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260928 20:23:30.703970 21741 catalog_manager.cc:4584] TS f5d77357f81a44d893b3a48164de1a62 (127.3.18.1:39625): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1f77dfbfc5114adaa729cf45f7698099: Network error: Client connection negotiation failed: client connection to 127.3.18.1:39625: connect: Connection refused (error 111)
I20260928 20:23:30.705569 22244 log.cc:1196] T 1f77dfbfc5114adaa729cf45f7698099 P a2001971f10b476d9c4ebe57d2989232: Deleting WAL directory at /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790626764250411-3144-0/minicluster-data/ts-2/wal/wals/1f77dfbfc5114adaa729cf45f7698099
I20260928 20:23:30.705873 22244 ts_tablet_manager.cc:1939] T 1f77dfbfc5114adaa729cf45f7698099 P a2001971f10b476d9c4ebe57d2989232: Deleting consensus metadata
I20260928 20:23:30.706377 21741 catalog_manager.cc:4841] TS a2001971f10b476d9c4ebe57d2989232 (127.3.18.3:45875): tablet 1f77dfbfc5114adaa729cf45f7698099 (table test-table [id=e2f44fc3473f4ed98f825af2a6957f8a]) successfully deleted
I20260928 20:23:30.708422 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 22078
I20260928 20:23:30.713384 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 21725
2026-09-28T20:23:30Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (776 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:23:30 dist-test-slave-b83v krb5kdc[22249](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:23:30 dist-test-slave-b83v krb5kdc[22249](info): set up 2 sockets
Sep 28 20:23:30 dist-test-slave-b83v krb5kdc[22249](info): commencing operation
krb5kdc: starting...
W20260928 20:23:31.812252 3144 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Sep 28 20:23:31 dist-test-slave-b83v krb5kdc[22249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:23:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:23:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.18.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab.
Entry for principal kudu/127.3.18.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab.
I20260928 20:23:31.953207 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:36903
--webserver_interface=127.3.18.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:34007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.18.62:36903
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:23:32.036728 22262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:23:32.036998 22262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:23:32.037039 22262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:23:32.038246 22262 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:23:32.038331 22262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:23:32.038355 22262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:23:32.038378 22262 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:23:32.038406 22262 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:23:32.039978 22262 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:34007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.18.62:36903
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:36903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.18.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:23:32.040282 22262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:23:32.040542 22262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:23:32.045818 22268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:32.045818 22267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:32.045820 22270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:23:32.045917 22262 server_base.cc:1034] running on GCE node
I20260928 20:23:32.046283 22262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:23:32.046595 22262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:23:32.047753 22262 hybrid_clock.cc:648] HybridClock initialized: now 1790627012047740 us; error 32 us; skew 500 ppm
Sep 28 20:23:32 dist-test-slave-b83v krb5kdc[22249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627012, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:23:32.050526 22262 init.cc:377] Logged in from keytab as kudu/127.3.18.62@KRBTEST.COM (short username kudu)
I20260928 20:23:32.051059 22262 webserver.cc:460] Webserver started at http://127.3.18.62:40623/ using document root <none> and password file <none>
I20260928 20:23:32.051321 22262 fs_manager.cc:359] Metadata directory not provided
I20260928 20:23:32.051366 22262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:23:32.051514 22262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:23:32.052695 22262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/minicluster-data/master-0/data/instance:
uuid: "fb09e1c928204107956de7463b62d1fc"
format_stamp: "Formatted at 2026-09-28 20:23:32 on dist-test-slave-b83v"
server_key: "ad055f9a3c4b5fce6a502d65d747fa31"
server_key_iv: "ae93592eb5dc31eca83b675ba2467108"
server_key_version: "encryptionkey@0"
I20260928 20:23:32.053077 22262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/minicluster-data/master-0/wal/instance:
uuid: "fb09e1c928204107956de7463b62d1fc"
format_stamp: "Formatted at 2026-09-28 20:23:32 on dist-test-slave-b83v"
server_key: "ad055f9a3c4b5fce6a502d65d747fa31"
server_key_iv: "ae93592eb5dc31eca83b675ba2467108"
server_key_version: "encryptionkey@0"
I20260928 20:23:32.054476 22262 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 20:23:32.055384 22277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:32.055641 22262 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:23:32.055750 22262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/minicluster-data/master-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/minicluster-data/master-0/wal
uuid: "fb09e1c928204107956de7463b62d1fc"
format_stamp: "Formatted at 2026-09-28 20:23:32 on dist-test-slave-b83v"
server_key: "ad055f9a3c4b5fce6a502d65d747fa31"
server_key_iv: "ae93592eb5dc31eca83b675ba2467108"
server_key_version: "encryptionkey@0"
I20260928 20:23:32.055855 22262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:23:32.064826 22262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:23:32.067330 22262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:23:32.067480 22262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:23:32.071811 22262 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.62:36903
I20260928 20:23:32.071909 22329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.62:36903 every 8 connection(s)
I20260928 20:23:32.072284 22262 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
I20260928 20:23:32.073264 22330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:23:32.075605 22330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb09e1c928204107956de7463b62d1fc: Bootstrap starting.
I20260928 20:23:32.076265 22330 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb09e1c928204107956de7463b62d1fc: Neither blocks nor log segments found. Creating new log.
I20260928 20:23:32.076602 22330 log.cc:824] T 00000000000000000000000000000000 P fb09e1c928204107956de7463b62d1fc: Log is configured to *not* fsync() on all Append() calls
I20260928 20:23:32.077407 22330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb09e1c928204107956de7463b62d1fc: No bootstrap required, opened a new log
I20260928 20:23:32.078536 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 22262
I20260928 20:23:32.078651 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/minicluster-data/master-0/wal/instance
I20260928 20:23:32.078889 3144 external_mini_cluster.cc:1428] Setting key 872f75b0166175e4407a074ffd6dd01b
I20260928 20:23:32.079439 22330 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb09e1c928204107956de7463b62d1fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb09e1c928204107956de7463b62d1fc" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36903 } }
I20260928 20:23:32.079659 22330 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb09e1c928204107956de7463b62d1fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:23:32.079713 22330 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb09e1c928204107956de7463b62d1fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb09e1c928204107956de7463b62d1fc, State: Initialized, Role: FOLLOWER
I20260928 20:23:32.079926 22330 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb09e1c928204107956de7463b62d1fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb09e1c928204107956de7463b62d1fc" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36903 } }
I20260928 20:23:32.080037 22330 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb09e1c928204107956de7463b62d1fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:23:32.080080 22330 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb09e1c928204107956de7463b62d1fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:23:32.080122 22330 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb09e1c928204107956de7463b62d1fc [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:23:32.081071 22330 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb09e1c928204107956de7463b62d1fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb09e1c928204107956de7463b62d1fc" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36903 } }
I20260928 20:23:32.081202 22330 leader_election.cc:302] T 00000000000000000000000000000000 P fb09e1c928204107956de7463b62d1fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb09e1c928204107956de7463b62d1fc; no voters:
I20260928 20:23:32.081383 22330 leader_election.cc:288] T 00000000000000000000000000000000 P fb09e1c928204107956de7463b62d1fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:23:32.081411 22335 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb09e1c928204107956de7463b62d1fc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:23:32.081517 22335 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb09e1c928204107956de7463b62d1fc [term 1 LEADER]: Becoming Leader. State: Replica: fb09e1c928204107956de7463b62d1fc, State: Running, Role: LEADER
I20260928 20:23:32.081607 22335 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb09e1c928204107956de7463b62d1fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb09e1c928204107956de7463b62d1fc" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36903 } }
I20260928 20:23:32.081669 22330 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb09e1c928204107956de7463b62d1fc [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:23:32.081935 22335 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb09e1c928204107956de7463b62d1fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb09e1c928204107956de7463b62d1fc. Latest consensus state: current_term: 1 leader_uuid: "fb09e1c928204107956de7463b62d1fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb09e1c928204107956de7463b62d1fc" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36903 } } }
I20260928 20:23:32.082006 22335 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb09e1c928204107956de7463b62d1fc [sys.catalog]: This master's current role is: LEADER
I20260928 20:23:32.082195 22336 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb09e1c928204107956de7463b62d1fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb09e1c928204107956de7463b62d1fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb09e1c928204107956de7463b62d1fc" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36903 } } }
I20260928 20:23:32.082293 22336 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb09e1c928204107956de7463b62d1fc [sys.catalog]: This master's current role is: LEADER
W20260928 20:23:32.083626 22351 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fb09e1c928204107956de7463b62d1fc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 20:23:32.083683 22351 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 20:23:32.083727 22340 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:23:32.083886 22340 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:23:32.085636 22340 catalog_manager.cc:1347] Generated new cluster ID: 74a10426220a4f12ab0c544795c75d27
I20260928 20:23:32.085693 22340 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 20:23:32 dist-test-slave-b83v krb5kdc[22249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:23:32.090219 22334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:32.079486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42858 (local address 127.3.18.62:36903)
0928 20:23:32.079672 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:32.079682 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:32.079729 (+ 47us) server_negotiation.cc:407] Connection header received
0928 20:23:32.079897 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:32.079903 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:32.079951 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:32.080106 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:32.080766 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:32.081988 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:32.083044 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:32.083582 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:32.083711 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:32.086570 (+ 2859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:32.086574 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:32.086574 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:23:32.086594 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:32.088408 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:32.088788 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:32.088792 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:32.088792 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:32.088912 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:32.089152 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:32.089153 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:32.089154 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:32.089421 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:32.089616 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:32.089689 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 20:23:32.089845 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":155,"threads_started":1}
I20260928 20:23:32.096983 22340 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:23:32.097653 22340 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:23:32.105298 22340 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb09e1c928204107956de7463b62d1fc: Generated new TSK 0
I20260928 20:23:32.105542 22340 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.18.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab.
Entry for principal kudu/127.3.18.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab.
I20260928 20:23:32.135854 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.1:0
--local_ip_for_outbound_sockets=127.3.18.1
--webserver_interface=127.3.18.1
--webserver_port=0
--tserver_master_addrs=127.3.18.62:36903
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab
--principal=kudu/127.3.18.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:34007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:23:32.222932 22356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:23:32.223165 22356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:23:32.223201 22356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:23:32.224514 22356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:23:32.224570 22356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:23:32.224598 22356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.1
I20260928 20:23:32.226200 22356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:34007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.1.keytab
--principal=kudu/127.3.18.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.18.1
--webserver_port=0
--tserver_master_addrs=127.3.18.62:36903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22356
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.1
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:23:32.226490 22356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:23:32.226780 22356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 20:23:32.232272 22356 server_base.cc:1034] running on GCE node
W20260928 20:23:32.232286 22364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:32.232275 22361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:32.232285 22362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:23:32.232831 22356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:23:32.233104 22356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:23:32.234295 22356 hybrid_clock.cc:648] HybridClock initialized: now 1790627012234281 us; error 31 us; skew 500 ppm
Sep 28 20:23:32 dist-test-slave-b83v krb5kdc[22249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627012, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:23:32.237051 22356 init.cc:377] Logged in from keytab as kudu/127.3.18.1@KRBTEST.COM (short username kudu)
I20260928 20:23:32.237521 22356 webserver.cc:460] Webserver started at http://127.3.18.1:43701/ using document root <none> and password file <none>
I20260928 20:23:32.237779 22356 fs_manager.cc:359] Metadata directory not provided
I20260928 20:23:32.237850 22356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:23:32.238003 22356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:23:32.239142 22356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/minicluster-data/ts-0/data/instance:
uuid: "3356fa82b0344c1ab2f75eb9929057a4"
format_stamp: "Formatted at 2026-09-28 20:23:32 on dist-test-slave-b83v"
server_key: "f3a8872ac3fa15e815e9daafce2fed0b"
server_key_iv: "a47600522f2a3c870c335cc0e9692a19"
server_key_version: "encryptionkey@0"
I20260928 20:23:32.239511 22356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/minicluster-data/ts-0/wal/instance:
uuid: "3356fa82b0344c1ab2f75eb9929057a4"
format_stamp: "Formatted at 2026-09-28 20:23:32 on dist-test-slave-b83v"
server_key: "f3a8872ac3fa15e815e9daafce2fed0b"
server_key_iv: "a47600522f2a3c870c335cc0e9692a19"
server_key_version: "encryptionkey@0"
I20260928 20:23:32.241070 22356 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 20:23:32.241954 22371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:32.242167 22356 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:23:32.242269 22356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/minicluster-data/ts-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/minicluster-data/ts-0/wal
uuid: "3356fa82b0344c1ab2f75eb9929057a4"
format_stamp: "Formatted at 2026-09-28 20:23:32 on dist-test-slave-b83v"
server_key: "f3a8872ac3fa15e815e9daafce2fed0b"
server_key_iv: "a47600522f2a3c870c335cc0e9692a19"
server_key_version: "encryptionkey@0"
I20260928 20:23:32.242358 22356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:23:32.250670 22356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:23:32.253188 22356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:23:32.253366 22356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:23:32.253629 22356 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:23:32.253980 22356 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:23:32.254042 22356 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:32.254088 22356 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:23:32.254112 22356 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:32.260326 22356 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.1:43999
I20260928 20:23:32.260401 22484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.1:43999 every 8 connection(s)
I20260928 20:23:32.260741 22356 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
I20260928 20:23:32.261564 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 22356
I20260928 20:23:32.261700 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/minicluster-data/ts-0/wal/instance
I20260928 20:23:32.261907 3144 external_mini_cluster.cc:1428] Setting key d982ad00e9d03fc23fc3f085e405c721
Sep 28 20:23:32 dist-test-slave-b83v krb5kdc[22249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627012, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.1@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.18.2@KRBTEST.COM; defaulting to no policy
I20260928 20:23:32.273764 22334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:32.261513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:55487 (local address 127.3.18.62:36903)
0928 20:23:32.261579 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:32.261582 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:32.261809 (+ 227us) server_negotiation.cc:407] Connection header received
0928 20:23:32.262534 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:32.262535 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:32.262551 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:32.262618 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:32.264058 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:32.264586 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:32.265760 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:32.266051 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:32.266144 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:32.269391 (+ 3247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:32.269392 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:32.269392 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:23:32.269400 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:32.271713 (+ 2313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:32.272442 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:32.272444 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:32.272444 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:32.272512 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:32.272876 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:32.272877 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:32.272877 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:32.273085 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:32.273154 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:32.273531 (+ 377us) server_negotiation.cc:299] Negotiation successful
0928 20:23:32.273669 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 20:23:32.273936 22487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:32.261512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36903 (local address 127.3.18.1:55487)
0928 20:23:32.261689 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:32.261725 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 20:23:32.262403 (+ 678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:23:32.262702 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:23:32.262706 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:23:32.263519 (+ 813us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:23:32.263933 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:32.263943 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:32.264695 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:32.264697 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:32.265616 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:32.265619 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:32.266182 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:32.266183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:32.266357 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:32.266942 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:23:32.266954 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:23:32.269232 (+ 2278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:23:32.271870 (+ 2638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:23:32.271882 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:23:32.271885 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:23:32.272329 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:23:32.272595 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:23:32.272596 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:23:32.272596 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:23:32.272785 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:23:32.273168 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:23:32.273177 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:23:32.273358 (+ 181us) client_negotiation.cc:769] Sending connection context
0928 20:23:32.273471 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 20:23:32.273674 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":90,"threads_started":1}
I20260928 20:23:32.274626 22485 heartbeater.cc:347] Connected to a master server at 127.3.18.62:36903
I20260928 20:23:32.274766 22485 heartbeater.cc:464] Registering TS with master...
I20260928 20:23:32.274997 22485 heartbeater.cc:511] Master 127.3.18.62:36903 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.2@KRBTEST.COM" created.
I20260928 20:23:32.276036 22294 ts_manager.cc:194] Registered new tserver with Master: 3356fa82b0344c1ab2f75eb9929057a4 (127.3.18.1:43999)
I20260928 20:23:32.277076 22294 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.18.1@KRBTEST.COM'} at 127.3.18.1:55487
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab.
Entry for principal kudu/127.3.18.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab.
I20260928 20:23:32.294929 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.2:0
--local_ip_for_outbound_sockets=127.3.18.2
--webserver_interface=127.3.18.2
--webserver_port=0
--tserver_master_addrs=127.3.18.62:36903
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab
--principal=kudu/127.3.18.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:34007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:23:32.377636 22490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:23:32.377866 22490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:23:32.377892 22490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:23:32.379007 22490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:23:32.379048 22490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:23:32.379074 22490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.2
I20260928 20:23:32.380523 22490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:34007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.2.keytab
--principal=kudu/127.3.18.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.18.2
--webserver_port=0
--tserver_master_addrs=127.3.18.62:36903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.2
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:23:32.380733 22490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:23:32.380913 22490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:23:32.386528 22496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:23:32.386583 22490 server_base.cc:1034] running on GCE node
W20260928 20:23:32.386539 22498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:32.386595 22495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:23:32.387006 22490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:23:32.387303 22490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:23:32.388501 22490 hybrid_clock.cc:648] HybridClock initialized: now 1790627012388489 us; error 27 us; skew 500 ppm
Sep 28 20:23:32 dist-test-slave-b83v krb5kdc[22249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627012, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:23:32.391127 22490 init.cc:377] Logged in from keytab as kudu/127.3.18.2@KRBTEST.COM (short username kudu)
I20260928 20:23:32.391649 22490 webserver.cc:460] Webserver started at http://127.3.18.2:45425/ using document root <none> and password file <none>
I20260928 20:23:32.391943 22490 fs_manager.cc:359] Metadata directory not provided
I20260928 20:23:32.392012 22490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:23:32.392159 22490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:23:32.393242 22490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/minicluster-data/ts-1/data/instance:
uuid: "cef7add054fd49358e7937866c090abb"
format_stamp: "Formatted at 2026-09-28 20:23:32 on dist-test-slave-b83v"
server_key: "88a40242c98f3e33f67cb3b58ce023b6"
server_key_iv: "424f2ef45f3ff4fe4e3a9d1fc5586ed1"
server_key_version: "encryptionkey@0"
I20260928 20:23:32.393635 22490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/minicluster-data/ts-1/wal/instance:
uuid: "cef7add054fd49358e7937866c090abb"
format_stamp: "Formatted at 2026-09-28 20:23:32 on dist-test-slave-b83v"
server_key: "88a40242c98f3e33f67cb3b58ce023b6"
server_key_iv: "424f2ef45f3ff4fe4e3a9d1fc5586ed1"
server_key_version: "encryptionkey@0"
I20260928 20:23:32.395083 22490 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:23:32.396006 22505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:32.396217 22490 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:23:32.396327 22490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/minicluster-data/ts-1/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/minicluster-data/ts-1/wal
uuid: "cef7add054fd49358e7937866c090abb"
format_stamp: "Formatted at 2026-09-28 20:23:32 on dist-test-slave-b83v"
server_key: "88a40242c98f3e33f67cb3b58ce023b6"
server_key_iv: "424f2ef45f3ff4fe4e3a9d1fc5586ed1"
server_key_version: "encryptionkey@0"
I20260928 20:23:32.396401 22490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:23:32.415904 22490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:23:32.418215 22490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:23:32.418421 22490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:23:32.418756 22490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:23:32.419121 22490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:23:32.419183 22490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:32.419231 22490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:23:32.419257 22490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:32.426029 22490 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.2:38889
I20260928 20:23:32.426084 22618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.2:38889 every 8 connection(s)
I20260928 20:23:32.426446 22490 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
I20260928 20:23:32.430348 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 22490
I20260928 20:23:32.430510 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/minicluster-data/ts-1/wal/instance
I20260928 20:23:32.430781 3144 external_mini_cluster.cc:1428] Setting key a28e2868e3a51419dc56999fa6ca099c
Sep 28 20:23:32 dist-test-slave-b83v krb5kdc[22249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627012, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.2@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:23:32.437013 22334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:32.427326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:46733 (local address 127.3.18.62:36903)
0928 20:23:32.427376 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:32.427378 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:32.427582 (+ 204us) server_negotiation.cc:407] Connection header received
0928 20:23:32.428065 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:32.428067 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:32.428077 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:32.428143 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:32.429142 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:32.429697 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:32.430962 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:32.431251 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:32.431311 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:32.433918 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:32.433921 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:32.433921 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:23:32.433928 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:32.435471 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:32.435995 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:32.435997 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:32.435997 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:32.436059 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:32.436333 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:32.436334 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:32.436334 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:32.436457 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:32.436536 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:32.436828 (+ 292us) server_negotiation.cc:299] Negotiation successful
0928 20:23:32.436918 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 20:23:32.437240 22621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:32.427242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36903 (local address 127.3.18.2:46733)
0928 20:23:32.427418 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:32.427461 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 20:23:32.427973 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:23:32.428215 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:23:32.428218 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:23:32.428822 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:23:32.429024 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:32.429029 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:32.429804 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:32.429805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:32.430834 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:32.430836 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:32.431337 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:32.431339 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:32.431500 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:32.432142 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:23:32.432155 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:23:32.433647 (+ 1492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:23:32.435571 (+ 1924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:23:32.435580 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:23:32.435583 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:23:32.435914 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:23:32.436127 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:23:32.436128 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:23:32.436128 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:23:32.436258 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:23:32.436525 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:23:32.436532 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:23:32.436690 (+ 158us) client_negotiation.cc:769] Sending connection context
0928 20:23:32.436796 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 20:23:32.436968 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":98,"threads_started":1}
I20260928 20:23:32.437953 22619 heartbeater.cc:347] Connected to a master server at 127.3.18.62:36903
I20260928 20:23:32.438052 22619 heartbeater.cc:464] Registering TS with master...
I20260928 20:23:32.438254 22619 heartbeater.cc:511] Master 127.3.18.62:36903 requested a full tablet report, sending...
I20260928 20:23:32.438666 22294 ts_manager.cc:194] Registered new tserver with Master: cef7add054fd49358e7937866c090abb (127.3.18.2:38889)
WARNING: no policy specified for kudu/127.3.18.3@KRBTEST.COM; defaulting to no policy
I20260928 20:23:32.439196 22294 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.18.2@KRBTEST.COM'} at 127.3.18.2:46733
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab.
Entry for principal kudu/127.3.18.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab.
I20260928 20:23:32.458779 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.3:0
--local_ip_for_outbound_sockets=127.3.18.3
--webserver_interface=127.3.18.3
--webserver_port=0
--tserver_master_addrs=127.3.18.62:36903
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab
--principal=kudu/127.3.18.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:34007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:23:32.541270 22624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:23:32.541483 22624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:23:32.541507 22624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:23:32.542688 22624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:23:32.542730 22624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:23:32.542757 22624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.3
I20260928 20:23:32.544301 22624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:34007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.3.keytab
--principal=kudu/127.3.18.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.18.3
--webserver_port=0
--tserver_master_addrs=127.3.18.62:36903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.3
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:23:32.544571 22624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:23:32.544833 22624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 20:23:32.550103 22624 server_base.cc:1034] running on GCE node
W20260928 20:23:32.550074 22632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:32.550086 22630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:32.550117 22629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:23:32.550559 22624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:23:32.550902 22624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:23:32.552075 22624 hybrid_clock.cc:648] HybridClock initialized: now 1790627012552020 us; error 74 us; skew 500 ppm
Sep 28 20:23:32 dist-test-slave-b83v krb5kdc[22249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627012, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:23:32.554720 22624 init.cc:377] Logged in from keytab as kudu/127.3.18.3@KRBTEST.COM (short username kudu)
I20260928 20:23:32.555217 22624 webserver.cc:460] Webserver started at http://127.3.18.3:45401/ using document root <none> and password file <none>
I20260928 20:23:32.555455 22624 fs_manager.cc:359] Metadata directory not provided
I20260928 20:23:32.555500 22624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:23:32.555595 22624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:23:32.556699 22624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/minicluster-data/ts-2/data/instance:
uuid: "96da0646b2434ccf810fc3c014755ca0"
format_stamp: "Formatted at 2026-09-28 20:23:32 on dist-test-slave-b83v"
server_key: "abd380cd1b01681985dbfe6d9db8ec6a"
server_key_iv: "0943e3fb2bd2c1c73789ad92227edded"
server_key_version: "encryptionkey@0"
I20260928 20:23:32.556998 22624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/minicluster-data/ts-2/wal/instance:
uuid: "96da0646b2434ccf810fc3c014755ca0"
format_stamp: "Formatted at 2026-09-28 20:23:32 on dist-test-slave-b83v"
server_key: "abd380cd1b01681985dbfe6d9db8ec6a"
server_key_iv: "0943e3fb2bd2c1c73789ad92227edded"
server_key_version: "encryptionkey@0"
I20260928 20:23:32.558367 22624 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 20:23:32.559140 22639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:32.559319 22624 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 20:23:32.559409 22624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/minicluster-data/ts-2/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/minicluster-data/ts-2/wal
uuid: "96da0646b2434ccf810fc3c014755ca0"
format_stamp: "Formatted at 2026-09-28 20:23:32 on dist-test-slave-b83v"
server_key: "abd380cd1b01681985dbfe6d9db8ec6a"
server_key_iv: "0943e3fb2bd2c1c73789ad92227edded"
server_key_version: "encryptionkey@0"
I20260928 20:23:32.559490 22624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:23:32.573452 22624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:23:32.576035 22624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:23:32.576217 22624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:23:32.576462 22624 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:23:32.576802 22624 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:23:32.576841 22624 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:32.576865 22624 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:23:32.576877 22624 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:32.583521 22624 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.3:34511
I20260928 20:23:32.583561 22752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.3:34511 every 8 connection(s)
I20260928 20:23:32.583977 22624 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
I20260928 20:23:32.584298 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 22624
I20260928 20:23:32.584439 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/minicluster-data/ts-2/wal/instance
I20260928 20:23:32.584671 3144 external_mini_cluster.cc:1428] Setting key 81f9aae7312b4233aff1d447b792c640
Sep 28 20:23:32 dist-test-slave-b83v krb5kdc[22249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627012, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.3@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:23:32.594413 22334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:32.584823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:36769 (local address 127.3.18.62:36903)
0928 20:23:32.584898 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:32.584900 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:32.585159 (+ 259us) server_negotiation.cc:407] Connection header received
0928 20:23:32.585909 (+ 750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:32.585910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:32.585923 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:32.585985 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:32.587061 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:32.587612 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:32.588786 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:32.588998 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:32.589085 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:32.591387 (+ 2302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:32.591389 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:32.591389 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:23:32.591397 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:32.592990 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:32.593425 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:32.593426 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:32.593426 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:32.593483 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:32.593788 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:32.593789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:32.593789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:32.593905 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:32.593969 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:32.594282 (+ 313us) server_negotiation.cc:299] Negotiation successful
0928 20:23:32.594346 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 20:23:32.594733 22755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:32.584788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36903 (local address 127.3.18.3:36769)
0928 20:23:32.584993 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:32.585070 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0928 20:23:32.585795 (+ 725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:23:32.586073 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:23:32.586077 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:23:32.586731 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:23:32.586942 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:32.586947 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:32.587715 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:32.587716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:32.588672 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:32.588674 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:32.589094 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:32.589095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:32.589248 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:32.589769 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:23:32.589779 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:23:32.591280 (+ 1501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:23:32.593097 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:23:32.593104 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:23:32.593106 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:23:32.593350 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:23:32.593580 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:23:32.593582 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:23:32.593582 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:23:32.593717 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:23:32.593974 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:23:32.593979 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:23:32.594154 (+ 175us) client_negotiation.cc:769] Sending connection context
0928 20:23:32.594257 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 20:23:32.594402 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":110,"threads_started":1}
I20260928 20:23:32.595346 22753 heartbeater.cc:347] Connected to a master server at 127.3.18.62:36903
I20260928 20:23:32.595443 22753 heartbeater.cc:464] Registering TS with master...
I20260928 20:23:32.595652 22753 heartbeater.cc:511] Master 127.3.18.62:36903 requested a full tablet report, sending...
I20260928 20:23:32.596163 22294 ts_manager.cc:194] Registered new tserver with Master: 96da0646b2434ccf810fc3c014755ca0 (127.3.18.3:34511)
I20260928 20:23:32.596609 3144 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 20:23:32.596704 22294 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.18.3@KRBTEST.COM'} at 127.3.18.3:36769
I20260928 20:23:32.604993 22334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:32.598063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33082 (local address 127.3.18.62:36903)
0928 20:23:32.598103 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:32.598105 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:32.598200 (+ 95us) server_negotiation.cc:407] Connection header received
0928 20:23:32.598267 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:32.598268 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:32.598276 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:32.598316 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:32.598874 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:32.599465 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:32.600477 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:32.600727 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:32.600798 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:32.602220 (+ 1422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:32.602221 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:32.602221 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:23:32.602228 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:32.603662 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:32.604133 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:32.604135 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:32.604135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:32.604193 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:32.604410 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:32.604411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:32.604412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:32.604598 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:32.604694 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:32.604843 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 20:23:32.604928 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 20:23:32.607088 22294 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33082:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 20:23:32.608870 22294 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 20:23:32.616561 22767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:32.612020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.3:34511 (local address 127.0.0.1:43732)
0928 20:23:32.612391 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:32.612399 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 20:23:32.612440 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:23:32.612861 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:23:32.612861 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:23:32.612864 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:23:32.612916 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:32.612918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:32.614200 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:32.614201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:32.615200 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:32.615203 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:32.616257 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:32.616258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:32.616348 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:32.616364 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 20:23:32.616448 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 20:23:32.616494 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":84,"threads_started":1}
I20260928 20:23:32.616804 22769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:32.612187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57980 (local address 127.3.18.2:38889)
0928 20:23:32.612540 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:32.612545 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:32.612554 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:23:32.612592 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:32.612594 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:32.612610 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:32.612697 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:23:32.612840 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:32.613630 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:32.615481 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:32.616273 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:32.616340 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:32.616401 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:32.616655 (+ 254us) server_negotiation.cc:299] Negotiation successful
0928 20:23:32.616711 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":76,"threads_started":1}
I20260928 20:23:32.616969 22768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:32.612167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43732 (local address 127.3.18.3:34511)
0928 20:23:32.612629 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:32.612635 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:32.612644 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:23:32.612679 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:32.612681 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:32.612696 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:32.612762 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:23:32.613004 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:32.614093 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:32.615313 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:32.616150 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:32.616239 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:32.616308 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:32.616482 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 20:23:32.616530 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":94,"threads_started":1}
I20260928 20:23:32.617395 22765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:32.611790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40718 (local address 127.3.18.1:43999)
0928 20:23:32.612002 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:32.612007 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:32.612037 (+ 30us) server_negotiation.cc:407] Connection header received
0928 20:23:32.612119 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:32.612122 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:32.612138 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:32.612259 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:23:32.612454 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:32.613735 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:32.615507 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:32.616148 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:32.616267 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:32.616334 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:32.617263 (+ 929us) server_negotiation.cc:299] Negotiation successful
0928 20:23:32.617308 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":90,"threads_started":1}
I20260928 20:23:32.616755 22766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:32.611839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.2:38889 (local address 127.0.0.1:57980)
0928 20:23:32.612351 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:32.612363 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:23:32.612419 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:23:32.612694 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:23:32.612695 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:23:32.612698 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:23:32.612745 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:32.612747 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:32.613733 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:32.613734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:32.615372 (+ 1638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:32.615376 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:32.616459 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:32.616461 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:32.616533 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:32.616537 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:23:32.616615 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 20:23:32.616668 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"mutex_wait_us":37,"thread_start_us":87,"threads_started":1}
I20260928 20:23:32.617355 22764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:32.611601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.1:43999 (local address 127.0.0.1:40718)
0928 20:23:32.611719 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:32.611783 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 20:23:32.611849 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:23:32.612265 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:23:32.612269 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:23:32.612271 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:23:32.612353 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:32.612355 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:32.613847 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:32.613847 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:32.615387 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:32.615390 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:32.617051 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:32.617052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:32.617134 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:32.617138 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:23:32.617239 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 20:23:32.617285 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"mutex_wait_us":79,"thread_start_us":73,"threads_started":1}
I20260928 20:23:32.618227 22553 tablet_service.cc:1467] Processing CreateTablet for tablet 02098e5b6525433b9a1d3d2755b752ef (DEFAULT_TABLE table=test-table [id=9df12f083dba4cff9002b69c5ff7498a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:23:32.618281 22687 tablet_service.cc:1467] Processing CreateTablet for tablet 02098e5b6525433b9a1d3d2755b752ef (DEFAULT_TABLE table=test-table [id=9df12f083dba4cff9002b69c5ff7498a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:23:32.618510 22553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02098e5b6525433b9a1d3d2755b752ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:23:32.618428 22419 tablet_service.cc:1467] Processing CreateTablet for tablet 02098e5b6525433b9a1d3d2755b752ef (DEFAULT_TABLE table=test-table [id=9df12f083dba4cff9002b69c5ff7498a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:23:32.618634 22687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02098e5b6525433b9a1d3d2755b752ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:23:32.618736 22419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02098e5b6525433b9a1d3d2755b752ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:23:32.621217 22770 tablet_bootstrap.cc:492] T 02098e5b6525433b9a1d3d2755b752ef P cef7add054fd49358e7937866c090abb: Bootstrap starting.
I20260928 20:23:32.621892 22770 tablet_bootstrap.cc:654] T 02098e5b6525433b9a1d3d2755b752ef P cef7add054fd49358e7937866c090abb: Neither blocks nor log segments found. Creating new log.
I20260928 20:23:32.622339 22771 tablet_bootstrap.cc:492] T 02098e5b6525433b9a1d3d2755b752ef P 3356fa82b0344c1ab2f75eb9929057a4: Bootstrap starting.
I20260928 20:23:32.622417 22770 log.cc:824] T 02098e5b6525433b9a1d3d2755b752ef P cef7add054fd49358e7937866c090abb: Log is configured to *not* fsync() on all Append() calls
I20260928 20:23:32.622720 22772 tablet_bootstrap.cc:492] T 02098e5b6525433b9a1d3d2755b752ef P 96da0646b2434ccf810fc3c014755ca0: Bootstrap starting.
I20260928 20:23:32.622985 22771 tablet_bootstrap.cc:654] T 02098e5b6525433b9a1d3d2755b752ef P 3356fa82b0344c1ab2f75eb9929057a4: Neither blocks nor log segments found. Creating new log.
I20260928 20:23:32.623299 22770 tablet_bootstrap.cc:492] T 02098e5b6525433b9a1d3d2755b752ef P cef7add054fd49358e7937866c090abb: No bootstrap required, opened a new log
I20260928 20:23:32.623389 22770 ts_tablet_manager.cc:1397] T 02098e5b6525433b9a1d3d2755b752ef P cef7add054fd49358e7937866c090abb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 20:23:32.623497 22772 tablet_bootstrap.cc:654] T 02098e5b6525433b9a1d3d2755b752ef P 96da0646b2434ccf810fc3c014755ca0: Neither blocks nor log segments found. Creating new log.
I20260928 20:23:32.623582 22771 log.cc:824] T 02098e5b6525433b9a1d3d2755b752ef P 3356fa82b0344c1ab2f75eb9929057a4: Log is configured to *not* fsync() on all Append() calls
I20260928 20:23:32.624158 22772 log.cc:824] T 02098e5b6525433b9a1d3d2755b752ef P 96da0646b2434ccf810fc3c014755ca0: Log is configured to *not* fsync() on all Append() calls
I20260928 20:23:32.624480 22771 tablet_bootstrap.cc:492] T 02098e5b6525433b9a1d3d2755b752ef P 3356fa82b0344c1ab2f75eb9929057a4: No bootstrap required, opened a new log
I20260928 20:23:32.624552 22771 ts_tablet_manager.cc:1397] T 02098e5b6525433b9a1d3d2755b752ef P 3356fa82b0344c1ab2f75eb9929057a4: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 20:23:32.624995 22772 tablet_bootstrap.cc:492] T 02098e5b6525433b9a1d3d2755b752ef P 96da0646b2434ccf810fc3c014755ca0: No bootstrap required, opened a new log
I20260928 20:23:32.625097 22772 ts_tablet_manager.cc:1397] T 02098e5b6525433b9a1d3d2755b752ef P 96da0646b2434ccf810fc3c014755ca0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 20:23:32.625411 22770 raft_consensus.cc:348] T 02098e5b6525433b9a1d3d2755b752ef P cef7add054fd49358e7937866c090abb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3356fa82b0344c1ab2f75eb9929057a4" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 43999 } } peers { permanent_uuid: "cef7add054fd49358e7937866c090abb" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 38889 } } peers { permanent_uuid: "96da0646b2434ccf810fc3c014755ca0" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 34511 } }
I20260928 20:23:32.625610 22770 raft_consensus.cc:374] T 02098e5b6525433b9a1d3d2755b752ef P cef7add054fd49358e7937866c090abb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:23:32.625658 22770 raft_consensus.cc:729] T 02098e5b6525433b9a1d3d2755b752ef P cef7add054fd49358e7937866c090abb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cef7add054fd49358e7937866c090abb, State: Initialized, Role: FOLLOWER
I20260928 20:23:32.625789 22770 consensus_queue.cc:260] T 02098e5b6525433b9a1d3d2755b752ef P cef7add054fd49358e7937866c090abb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3356fa82b0344c1ab2f75eb9929057a4" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 43999 } } peers { permanent_uuid: "cef7add054fd49358e7937866c090abb" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 38889 } } peers { permanent_uuid: "96da0646b2434ccf810fc3c014755ca0" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 34511 } }
I20260928 20:23:32.626061 22770 ts_tablet_manager.cc:1428] T 02098e5b6525433b9a1d3d2755b752ef P cef7add054fd49358e7937866c090abb: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 20:23:32.626206 22619 heartbeater.cc:503] Master 127.3.18.62:36903 was elected leader, sending a full tablet report...
I20260928 20:23:32.626694 22771 raft_consensus.cc:348] T 02098e5b6525433b9a1d3d2755b752ef P 3356fa82b0344c1ab2f75eb9929057a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3356fa82b0344c1ab2f75eb9929057a4" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 43999 } } peers { permanent_uuid: "cef7add054fd49358e7937866c090abb" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 38889 } } peers { permanent_uuid: "96da0646b2434ccf810fc3c014755ca0" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 34511 } }
I20260928 20:23:32.626856 22771 raft_consensus.cc:374] T 02098e5b6525433b9a1d3d2755b752ef P 3356fa82b0344c1ab2f75eb9929057a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:23:32.626899 22771 raft_consensus.cc:729] T 02098e5b6525433b9a1d3d2755b752ef P 3356fa82b0344c1ab2f75eb9929057a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3356fa82b0344c1ab2f75eb9929057a4, State: Initialized, Role: FOLLOWER
I20260928 20:23:32.627022 22771 consensus_queue.cc:260] T 02098e5b6525433b9a1d3d2755b752ef P 3356fa82b0344c1ab2f75eb9929057a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3356fa82b0344c1ab2f75eb9929057a4" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 43999 } } peers { permanent_uuid: "cef7add054fd49358e7937866c090abb" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 38889 } } peers { permanent_uuid: "96da0646b2434ccf810fc3c014755ca0" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 34511 } }
I20260928 20:23:32.627310 22771 ts_tablet_manager.cc:1428] T 02098e5b6525433b9a1d3d2755b752ef P 3356fa82b0344c1ab2f75eb9929057a4: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 20:23:32.627367 22485 heartbeater.cc:503] Master 127.3.18.62:36903 was elected leader, sending a full tablet report...
I20260928 20:23:32.627549 22772 raft_consensus.cc:348] T 02098e5b6525433b9a1d3d2755b752ef P 96da0646b2434ccf810fc3c014755ca0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3356fa82b0344c1ab2f75eb9929057a4" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 43999 } } peers { permanent_uuid: "cef7add054fd49358e7937866c090abb" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 38889 } } peers { permanent_uuid: "96da0646b2434ccf810fc3c014755ca0" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 34511 } }
I20260928 20:23:32.627737 22772 raft_consensus.cc:374] T 02098e5b6525433b9a1d3d2755b752ef P 96da0646b2434ccf810fc3c014755ca0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:23:32.627825 22772 raft_consensus.cc:729] T 02098e5b6525433b9a1d3d2755b752ef P 96da0646b2434ccf810fc3c014755ca0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96da0646b2434ccf810fc3c014755ca0, State: Initialized, Role: FOLLOWER
I20260928 20:23:32.627962 22772 consensus_queue.cc:260] T 02098e5b6525433b9a1d3d2755b752ef P 96da0646b2434ccf810fc3c014755ca0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3356fa82b0344c1ab2f75eb9929057a4" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 43999 } } peers { permanent_uuid: "cef7add054fd49358e7937866c090abb" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 38889 } } peers { permanent_uuid: "96da0646b2434ccf810fc3c014755ca0" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 34511 } }
I20260928 20:23:32.628310 22753 heartbeater.cc:503] Master 127.3.18.62:36903 was elected leader, sending a full tablet report...
I20260928 20:23:32.628309 22772 ts_tablet_manager.cc:1428] T 02098e5b6525433b9a1d3d2755b752ef P 96da0646b2434ccf810fc3c014755ca0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260928 20:23:32.677096 22620 tablet.cc:2369] T 02098e5b6525433b9a1d3d2755b752ef P cef7add054fd49358e7937866c090abb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 20:23:32.692608 22777 raft_consensus.cc:482] T 02098e5b6525433b9a1d3d2755b752ef P 3356fa82b0344c1ab2f75eb9929057a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:23:32.692749 22777 raft_consensus.cc:504] T 02098e5b6525433b9a1d3d2755b752ef P 3356fa82b0344c1ab2f75eb9929057a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3356fa82b0344c1ab2f75eb9929057a4" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 43999 } } peers { permanent_uuid: "cef7add054fd49358e7937866c090abb" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 38889 } } peers { permanent_uuid: "96da0646b2434ccf810fc3c014755ca0" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 34511 } }
I20260928 20:23:32.693188 22777 leader_election.cc:288] T 02098e5b6525433b9a1d3d2755b752ef P 3356fa82b0344c1ab2f75eb9929057a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cef7add054fd49358e7937866c090abb (127.3.18.2:38889), 96da0646b2434ccf810fc3c014755ca0 (127.3.18.3:34511)
I20260928 20:23:32.696722 22768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:32.693756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:59727 (local address 127.3.18.3:34511)
0928 20:23:32.693807 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:32.693809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:32.693818 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:23:32.693893 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:32.693894 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:32.693907 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:32.693993 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:23:32.694178 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:32.694659 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:32.695961 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:32.696329 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:32.696437 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:32.696448 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:32.696600 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 20:23:32.696648 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 20:23:32.696803 22769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:32.693524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:53849 (local address 127.3.18.2:38889)
0928 20:23:32.693585 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:32.693587 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:32.693605 (+ 18us) server_negotiation.cc:407] Connection header received
0928 20:23:32.693633 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:32.693634 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:32.693645 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:32.693748 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:23:32.693964 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:32.694728 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:32.695885 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:32.696332 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:32.696426 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:32.696438 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:32.696690 (+ 252us) server_negotiation.cc:299] Negotiation successful
0928 20:23:32.696733 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 20:23:32.696902 22487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:32.693439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.2:38889 (local address 127.3.18.1:53849)
0928 20:23:32.693505 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:32.693518 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 20:23:32.693568 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:23:32.693781 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:23:32.693783 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:23:32.693785 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:23:32.693863 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:32.693865 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:32.694859 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:32.694861 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:32.695719 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:32.695722 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:32.696426 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:32.696428 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:32.696600 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:32.696605 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 20:23:32.696669 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 20:23:32.696710 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":27}
I20260928 20:23:32.696730 22779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:32.693591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.3:34511 (local address 127.3.18.1:59727)
0928 20:23:32.693771 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:32.693781 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 20:23:32.693826 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:23:32.694018 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:23:32.694019 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:23:32.694021 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:23:32.694099 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:32.694101 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:32.694759 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:32.694761 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:32.695867 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:32.695871 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:32.696444 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:32.696445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:32.696523 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:32.696526 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 20:23:32.696575 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 20:23:32.696615 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":72,"thread_start_us":117,"threads_started":1}
I20260928 20:23:32.697338 22573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02098e5b6525433b9a1d3d2755b752ef" candidate_uuid: "3356fa82b0344c1ab2f75eb9929057a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cef7add054fd49358e7937866c090abb" is_pre_election: true
I20260928 20:23:32.697448 22707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02098e5b6525433b9a1d3d2755b752ef" candidate_uuid: "3356fa82b0344c1ab2f75eb9929057a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96da0646b2434ccf810fc3c014755ca0" is_pre_election: true
I20260928 20:23:32.697523 22573 raft_consensus.cc:2454] T 02098e5b6525433b9a1d3d2755b752ef P cef7add054fd49358e7937866c090abb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3356fa82b0344c1ab2f75eb9929057a4 in term 0.
I20260928 20:23:32.697644 22707 raft_consensus.cc:2454] T 02098e5b6525433b9a1d3d2755b752ef P 96da0646b2434ccf810fc3c014755ca0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3356fa82b0344c1ab2f75eb9929057a4 in term 0.
I20260928 20:23:32.697794 22375 leader_election.cc:302] T 02098e5b6525433b9a1d3d2755b752ef P 3356fa82b0344c1ab2f75eb9929057a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3356fa82b0344c1ab2f75eb9929057a4, cef7add054fd49358e7937866c090abb; no voters:
I20260928 20:23:32.697953 22777 raft_consensus.cc:2781] T 02098e5b6525433b9a1d3d2755b752ef P 3356fa82b0344c1ab2f75eb9929057a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 20:23:32.698017 22777 raft_consensus.cc:482] T 02098e5b6525433b9a1d3d2755b752ef P 3356fa82b0344c1ab2f75eb9929057a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 20:23:32.698066 22777 raft_consensus.cc:3037] T 02098e5b6525433b9a1d3d2755b752ef P 3356fa82b0344c1ab2f75eb9929057a4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:23:32.698904 22777 raft_consensus.cc:504] T 02098e5b6525433b9a1d3d2755b752ef P 3356fa82b0344c1ab2f75eb9929057a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3356fa82b0344c1ab2f75eb9929057a4" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 43999 } } peers { permanent_uuid: "cef7add054fd49358e7937866c090abb" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 38889 } } peers { permanent_uuid: "96da0646b2434ccf810fc3c014755ca0" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 34511 } }
I20260928 20:23:32.699071 22777 leader_election.cc:288] T 02098e5b6525433b9a1d3d2755b752ef P 3356fa82b0344c1ab2f75eb9929057a4 [CANDIDATE]: Term 1 election: Requested vote from peers cef7add054fd49358e7937866c090abb (127.3.18.2:38889), 96da0646b2434ccf810fc3c014755ca0 (127.3.18.3:34511)
I20260928 20:23:32.699267 22707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02098e5b6525433b9a1d3d2755b752ef" candidate_uuid: "3356fa82b0344c1ab2f75eb9929057a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96da0646b2434ccf810fc3c014755ca0"
I20260928 20:23:32.699239 22573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02098e5b6525433b9a1d3d2755b752ef" candidate_uuid: "3356fa82b0344c1ab2f75eb9929057a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cef7add054fd49358e7937866c090abb"
I20260928 20:23:32.699340 22707 raft_consensus.cc:3037] T 02098e5b6525433b9a1d3d2755b752ef P 96da0646b2434ccf810fc3c014755ca0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:23:32.699348 22573 raft_consensus.cc:3037] T 02098e5b6525433b9a1d3d2755b752ef P cef7add054fd49358e7937866c090abb [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:23:32.700335 22573 raft_consensus.cc:2454] T 02098e5b6525433b9a1d3d2755b752ef P cef7add054fd49358e7937866c090abb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3356fa82b0344c1ab2f75eb9929057a4 in term 1.
I20260928 20:23:32.700335 22707 raft_consensus.cc:2454] T 02098e5b6525433b9a1d3d2755b752ef P 96da0646b2434ccf810fc3c014755ca0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3356fa82b0344c1ab2f75eb9929057a4 in term 1.
I20260928 20:23:32.700541 22375 leader_election.cc:302] T 02098e5b6525433b9a1d3d2755b752ef P 3356fa82b0344c1ab2f75eb9929057a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3356fa82b0344c1ab2f75eb9929057a4, 96da0646b2434ccf810fc3c014755ca0; no voters:
I20260928 20:23:32.700672 22777 raft_consensus.cc:2781] T 02098e5b6525433b9a1d3d2755b752ef P 3356fa82b0344c1ab2f75eb9929057a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:23:32.700862 22777 raft_consensus.cc:686] T 02098e5b6525433b9a1d3d2755b752ef P 3356fa82b0344c1ab2f75eb9929057a4 [term 1 LEADER]: Becoming Leader. State: Replica: 3356fa82b0344c1ab2f75eb9929057a4, State: Running, Role: LEADER
I20260928 20:23:32.700977 22777 consensus_queue.cc:237] T 02098e5b6525433b9a1d3d2755b752ef P 3356fa82b0344c1ab2f75eb9929057a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3356fa82b0344c1ab2f75eb9929057a4" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 43999 } } peers { permanent_uuid: "cef7add054fd49358e7937866c090abb" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 38889 } } peers { permanent_uuid: "96da0646b2434ccf810fc3c014755ca0" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 34511 } }
I20260928 20:23:32.701911 22293 catalog_manager.cc:5495] T 02098e5b6525433b9a1d3d2755b752ef P 3356fa82b0344c1ab2f75eb9929057a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3356fa82b0344c1ab2f75eb9929057a4 (127.3.18.1). New cstate: current_term: 1 leader_uuid: "3356fa82b0344c1ab2f75eb9929057a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3356fa82b0344c1ab2f75eb9929057a4" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 43999 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cef7add054fd49358e7937866c090abb" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 38889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96da0646b2434ccf810fc3c014755ca0" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 34511 } health_report { overall_health: UNKNOWN } } }
I20260928 20:23:32.710083 22765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:32.707386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40734 (local address 127.3.18.1:43999)
0928 20:23:32.707439 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:32.707441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:32.707450 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:23:32.707624 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:32.707625 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:32.707655 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:32.707746 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:23:32.707940 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:32.708426 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:32.709255 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:32.709459 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:32.709551 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:32.709732 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:23:32.709841 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:23:32.709945 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:32.709969 (+ 24us) server_negotiation.cc:299] Negotiation successful
0928 20:23:32.709992 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 20:23:32.712170 22707 raft_consensus.cc:1261] T 02098e5b6525433b9a1d3d2755b752ef P 96da0646b2434ccf810fc3c014755ca0 [term 1 FOLLOWER]: Refusing update from remote peer 3356fa82b0344c1ab2f75eb9929057a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:23:32.712208 22573 raft_consensus.cc:1261] T 02098e5b6525433b9a1d3d2755b752ef P cef7add054fd49358e7937866c090abb [term 1 FOLLOWER]: Refusing update from remote peer 3356fa82b0344c1ab2f75eb9929057a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:23:32.712591 22777 consensus_queue.cc:1035] T 02098e5b6525433b9a1d3d2755b752ef P 3356fa82b0344c1ab2f75eb9929057a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cef7add054fd49358e7937866c090abb" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:23:32.712764 22780 consensus_queue.cc:1035] T 02098e5b6525433b9a1d3d2755b752ef P 3356fa82b0344c1ab2f75eb9929057a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96da0646b2434ccf810fc3c014755ca0" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 34511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:23:32.715849 22785 mvcc.cc:205] Tried to move back new op lower bound from 7334408244065464320 to 7334408244023971840. Current Snapshot: MvccSnapshot[applied={T|T < 7334408244065464320}]
I20260928 20:23:32.716554 22784 mvcc.cc:205] Tried to move back new op lower bound from 7334408244065464320 to 7334408244023971840. Current Snapshot: MvccSnapshot[applied={T|T < 7334408244065464320 or (T in {7334408244065464320})}]
I20260928 20:23:32.717482 22293 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:23:32.717574 22293 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:23:32.718339 22293 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fb09e1c928204107956de7463b62d1fc: Sending DeleteTablet for 3 replicas of tablet 02098e5b6525433b9a1d3d2755b752ef
I20260928 20:23:32.718712 22419 tablet_service.cc:1514] Processing DeleteTablet for tablet 02098e5b6525433b9a1d3d2755b752ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:23:32 UTC) from {username='kudu', principal='kudu/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:40718
I20260928 20:23:32.718714 22553 tablet_service.cc:1514] Processing DeleteTablet for tablet 02098e5b6525433b9a1d3d2755b752ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:23:32 UTC) from {username='kudu', principal='kudu/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:57980
I20260928 20:23:32.718945 22793 tablet_replica.cc:330] T 02098e5b6525433b9a1d3d2755b752ef P 3356fa82b0344c1ab2f75eb9929057a4: stopping tablet replica
I20260928 20:23:32.718930 22687 tablet_service.cc:1514] Processing DeleteTablet for tablet 02098e5b6525433b9a1d3d2755b752ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:23:32 UTC) from {username='kudu', principal='kudu/127.3.18.62@KRBTEST.COM'} at 127.0.0.1:43732
I20260928 20:23:32.719056 22793 raft_consensus.cc:2229] T 02098e5b6525433b9a1d3d2755b752ef P 3356fa82b0344c1ab2f75eb9929057a4 [term 1 LEADER]: Raft consensus shutting down.
I20260928 20:23:32.719164 22794 tablet_replica.cc:330] T 02098e5b6525433b9a1d3d2755b752ef P 96da0646b2434ccf810fc3c014755ca0: stopping tablet replica
I20260928 20:23:32.719192 22793 raft_consensus.cc:2258] T 02098e5b6525433b9a1d3d2755b752ef P 3356fa82b0344c1ab2f75eb9929057a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:23:32.719372 22792 tablet_replica.cc:330] T 02098e5b6525433b9a1d3d2755b752ef P cef7add054fd49358e7937866c090abb: stopping tablet replica
I20260928 20:23:32.719468 22792 raft_consensus.cc:2229] T 02098e5b6525433b9a1d3d2755b752ef P cef7add054fd49358e7937866c090abb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:23:32.719560 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 22356
I20260928 20:23:32.719606 22792 raft_consensus.cc:2258] T 02098e5b6525433b9a1d3d2755b752ef P cef7add054fd49358e7937866c090abb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:23:32.721124 22792 ts_tablet_manager.cc:1905] T 02098e5b6525433b9a1d3d2755b752ef P cef7add054fd49358e7937866c090abb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:23:32.721990 22794 raft_consensus.cc:2229] T 02098e5b6525433b9a1d3d2755b752ef P 96da0646b2434ccf810fc3c014755ca0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:23:32.722134 22794 raft_consensus.cc:2258] T 02098e5b6525433b9a1d3d2755b752ef P 96da0646b2434ccf810fc3c014755ca0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:23:32.722592 22794 ts_tablet_manager.cc:1905] T 02098e5b6525433b9a1d3d2755b752ef P 96da0646b2434ccf810fc3c014755ca0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:23:32.722790 22792 ts_tablet_manager.cc:1918] T 02098e5b6525433b9a1d3d2755b752ef P cef7add054fd49358e7937866c090abb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:23:32.722874 22792 log.cc:1196] T 02098e5b6525433b9a1d3d2755b752ef P cef7add054fd49358e7937866c090abb: Deleting WAL directory at /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/minicluster-data/ts-1/wal/wals/02098e5b6525433b9a1d3d2755b752ef
I20260928 20:23:32.723125 22792 ts_tablet_manager.cc:1939] T 02098e5b6525433b9a1d3d2755b752ef P cef7add054fd49358e7937866c090abb: Deleting consensus metadata
I20260928 20:23:32.723666 22281 catalog_manager.cc:4841] TS cef7add054fd49358e7937866c090abb (127.3.18.2:38889): tablet 02098e5b6525433b9a1d3d2755b752ef (table test-table [id=9df12f083dba4cff9002b69c5ff7498a]) successfully deleted
I20260928 20:23:32.724540 22794 ts_tablet_manager.cc:1918] T 02098e5b6525433b9a1d3d2755b752ef P 96da0646b2434ccf810fc3c014755ca0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:23:32.724622 22794 log.cc:1196] T 02098e5b6525433b9a1d3d2755b752ef P 96da0646b2434ccf810fc3c014755ca0: Deleting WAL directory at /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790626764250411-3144-0/minicluster-data/ts-2/wal/wals/02098e5b6525433b9a1d3d2755b752ef
I20260928 20:23:32.724872 22794 ts_tablet_manager.cc:1939] T 02098e5b6525433b9a1d3d2755b752ef P 96da0646b2434ccf810fc3c014755ca0: Deleting consensus metadata
I20260928 20:23:32.725323 22281 catalog_manager.cc:4841] TS 96da0646b2434ccf810fc3c014755ca0 (127.3.18.3:34511): tablet 02098e5b6525433b9a1d3d2755b752ef (table test-table [id=9df12f083dba4cff9002b69c5ff7498a]) successfully deleted
W20260928 20:23:32.726181 22281 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.18.1:43999 (error 108)
I20260928 20:23:32.726318 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 22490
I20260928 20:23:32.727468 22764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:32.727313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.1:43999 (local address 127.0.0.1:40736)
0928 20:23:32.727380 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:32.727425 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.1:43999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 20:23:32.727628 22281 catalog_manager.cc:4584] TS 3356fa82b0344c1ab2f75eb9929057a4 (127.3.18.1:43999): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 02098e5b6525433b9a1d3d2755b752ef: Network error: Client connection negotiation failed: client connection to 127.3.18.1:43999: connect: Connection refused (error 111)
I20260928 20:23:32.731391 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 22624
I20260928 20:23:32.736158 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 22262
2026-09-28T20:23:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2025 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T20:23:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:23:32Z Disabled control of system clock
I20260928 20:23:32.766603 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790626764250411-3144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790626764250411-3144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790626764250411-3144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:39971
--webserver_interface=127.3.18.62
--webserver_port=0
--builtin_ntp_servers=127.3.18.20:33007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.18.62:39971
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 20:23:32.847851 22799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:23:32.848080 22799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:23:32.848107 22799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:23:32.849237 22799 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:23:32.849275 22799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:23:32.849288 22799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:23:32.849300 22799 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:23:32.849313 22799 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:23:32.850688 22799 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:33007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790626764250411-3144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790626764250411-3144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.18.62:39971
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:39971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.18.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790626764250411-3144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:23:32.850889 22799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:23:32.851140 22799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:23:32.856493 22804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:23:32.856524 22799 server_base.cc:1034] running on GCE node
W20260928 20:23:32.856524 22805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:32.856493 22807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:23:32.857189 22799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:23:32.857476 22799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:23:32.858628 22799 hybrid_clock.cc:648] HybridClock initialized: now 1790627012858612 us; error 31 us; skew 500 ppm
I20260928 20:23:32.860172 22799 webserver.cc:460] Webserver started at http://127.3.18.62:34033/ using document root <none> and password file <none>
I20260928 20:23:32.860442 22799 fs_manager.cc:359] Metadata directory not provided
I20260928 20:23:32.860495 22799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:23:32.860636 22799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:23:32.861748 22799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790626764250411-3144-0/minicluster-data/master-0/data/instance:
uuid: "b5d22b2c396d438880b46f12d5580558"
format_stamp: "Formatted at 2026-09-28 20:23:32 on dist-test-slave-b83v"
server_key: "de7b0df3642822caf093c73131f748e0"
server_key_iv: "2718934a008086e1114ec6a1f1ce1a53"
server_key_version: "encryptionkey@0"
I20260928 20:23:32.862118 22799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790626764250411-3144-0/minicluster-data/master-0/wal/instance:
uuid: "b5d22b2c396d438880b46f12d5580558"
format_stamp: "Formatted at 2026-09-28 20:23:32 on dist-test-slave-b83v"
server_key: "de7b0df3642822caf093c73131f748e0"
server_key_iv: "2718934a008086e1114ec6a1f1ce1a53"
server_key_version: "encryptionkey@0"
I20260928 20:23:32.863449 22799 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:23:32.864307 22813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:32.864485 22799 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 20:23:32.864598 22799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790626764250411-3144-0/minicluster-data/master-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790626764250411-3144-0/minicluster-data/master-0/wal
uuid: "b5d22b2c396d438880b46f12d5580558"
format_stamp: "Formatted at 2026-09-28 20:23:32 on dist-test-slave-b83v"
server_key: "de7b0df3642822caf093c73131f748e0"
server_key_iv: "2718934a008086e1114ec6a1f1ce1a53"
server_key_version: "encryptionkey@0"
I20260928 20:23:32.864679 22799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790626764250411-3144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790626764250411-3144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790626764250411-3144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:23:32.881301 22799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:23:32.881682 22799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:23:32.881844 22799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:23:32.886034 22799 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.62:39971
I20260928 20:23:32.886071 22865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.62:39971 every 8 connection(s)
I20260928 20:23:32.886438 22799 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
I20260928 20:23:32.887451 22866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:23:32.889889 22866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5d22b2c396d438880b46f12d5580558: Bootstrap starting.
I20260928 20:23:32.890491 22866 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5d22b2c396d438880b46f12d5580558: Neither blocks nor log segments found. Creating new log.
I20260928 20:23:32.890808 22866 log.cc:824] T 00000000000000000000000000000000 P b5d22b2c396d438880b46f12d5580558: Log is configured to *not* fsync() on all Append() calls
I20260928 20:23:32.891206 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 22799
I20260928 20:23:32.891360 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790626764250411-3144-0/minicluster-data/master-0/wal/instance
I20260928 20:23:32.891584 3144 external_mini_cluster.cc:1428] Setting key f45127d94e0208e0dab9ed1b1bdd62ca
I20260928 20:23:32.891580 22866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5d22b2c396d438880b46f12d5580558: No bootstrap required, opened a new log
I20260928 20:23:32.894109 22866 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5d22b2c396d438880b46f12d5580558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5d22b2c396d438880b46f12d5580558" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 39971 } }
I20260928 20:23:32.894330 22866 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5d22b2c396d438880b46f12d5580558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:23:32.894371 22866 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5d22b2c396d438880b46f12d5580558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5d22b2c396d438880b46f12d5580558, State: Initialized, Role: FOLLOWER
I20260928 20:23:32.894550 22866 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5d22b2c396d438880b46f12d5580558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5d22b2c396d438880b46f12d5580558" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 39971 } }
I20260928 20:23:32.894649 22866 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5d22b2c396d438880b46f12d5580558 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:23:32.894680 22866 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5d22b2c396d438880b46f12d5580558 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:23:32.894718 22866 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5d22b2c396d438880b46f12d5580558 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:23:32.895432 22866 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5d22b2c396d438880b46f12d5580558 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5d22b2c396d438880b46f12d5580558" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 39971 } }
I20260928 20:23:32.895548 22866 leader_election.cc:302] T 00000000000000000000000000000000 P b5d22b2c396d438880b46f12d5580558 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5d22b2c396d438880b46f12d5580558; no voters:
I20260928 20:23:32.895737 22866 leader_election.cc:288] T 00000000000000000000000000000000 P b5d22b2c396d438880b46f12d5580558 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:23:32.895818 22871 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5d22b2c396d438880b46f12d5580558 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:23:32.895941 22871 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5d22b2c396d438880b46f12d5580558 [term 1 LEADER]: Becoming Leader. State: Replica: b5d22b2c396d438880b46f12d5580558, State: Running, Role: LEADER
I20260928 20:23:32.896044 22866 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5d22b2c396d438880b46f12d5580558 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:23:32.896023 22871 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5d22b2c396d438880b46f12d5580558 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5d22b2c396d438880b46f12d5580558" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 39971 } }
I20260928 20:23:32.896487 22871 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5d22b2c396d438880b46f12d5580558 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5d22b2c396d438880b46f12d5580558. Latest consensus state: current_term: 1 leader_uuid: "b5d22b2c396d438880b46f12d5580558" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5d22b2c396d438880b46f12d5580558" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 39971 } } }
I20260928 20:23:32.896595 22871 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5d22b2c396d438880b46f12d5580558 [sys.catalog]: This master's current role is: LEADER
I20260928 20:23:32.896500 22873 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5d22b2c396d438880b46f12d5580558 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5d22b2c396d438880b46f12d5580558" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5d22b2c396d438880b46f12d5580558" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 39971 } } }
I20260928 20:23:32.896999 22873 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5d22b2c396d438880b46f12d5580558 [sys.catalog]: This master's current role is: LEADER
W20260928 20:23:32.898242 22887 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b5d22b2c396d438880b46f12d5580558: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 20:23:32.898307 22887 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 20:23:32.898357 22878 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:23:32.898524 22878 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:23:32.899328 22870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:32.892206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36010 (local address 127.3.18.62:39971)
0928 20:23:32.892427 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:32.892448 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:32.892489 (+ 41us) server_negotiation.cc:407] Connection header received
0928 20:23:32.893121 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:32.893131 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:32.893212 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:32.893370 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:32.893605 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:32.895467 (+ 1862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:32.896654 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:32.897314 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:32.897415 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:32.898573 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:32.898577 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:32.898577 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:23:32.898780 (+ 203us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:32.898813 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:23:32.898824 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:32.898898 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:32.898982 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 20:23:32.899069 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":85,"threads_started":1}
I20260928 20:23:32.900153 22878 catalog_manager.cc:1347] Generated new cluster ID: b5568f9847f94272a389d3069e2b4003
I20260928 20:23:32.900269 22878 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:23:32.948752 22878 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:23:32.949394 22878 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:23:32.966110 22878 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5d22b2c396d438880b46f12d5580558: Generated new TSK 0
I20260928 20:23:32.966327 22878 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 20:23:32.972889 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790626764250411-3144-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790626764250411-3144-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790626764250411-3144-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.1:0
--local_ip_for_outbound_sockets=127.3.18.1
--webserver_interface=127.3.18.1
--webserver_port=0
--tserver_master_addrs=127.3.18.62:39971
--builtin_ntp_servers=127.3.18.20:33007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 20:23:33.075446 22890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:23:33.075670 22890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:23:33.075695 22890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:23:33.076879 22890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:23:33.076923 22890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:23:33.076948 22890 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.1
I20260928 20:23:33.078254 22890 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:33007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790626764250411-3144-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790626764250411-3144-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.18.1
--webserver_port=0
--tserver_master_addrs=127.3.18.62:39971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.1
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790626764250411-3144-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:23:33.078462 22890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:23:33.078720 22890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:23:33.084360 22896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:33.084342 22895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:33.084342 22898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:23:33.084374 22890 server_base.cc:1034] running on GCE node
I20260928 20:23:33.084915 22890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:23:33.085213 22890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:23:33.086404 22890 hybrid_clock.cc:648] HybridClock initialized: now 1790627013086389 us; error 31 us; skew 500 ppm
I20260928 20:23:33.088055 22890 webserver.cc:460] Webserver started at http://127.3.18.1:38805/ using document root <none> and password file <none>
I20260928 20:23:33.088339 22890 fs_manager.cc:359] Metadata directory not provided
I20260928 20:23:33.088406 22890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:23:33.088573 22890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:23:33.089696 22890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790626764250411-3144-0/minicluster-data/ts-0/data/instance:
uuid: "af19c8a20ffb4b92841fd89231c562ba"
format_stamp: "Formatted at 2026-09-28 20:23:33 on dist-test-slave-b83v"
server_key: "6fd6ccf694a20943a12ebcef9f9d8cef"
server_key_iv: "a5131792c45b59a87fe54b9115f44b2b"
server_key_version: "encryptionkey@0"
I20260928 20:23:33.090054 22890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790626764250411-3144-0/minicluster-data/ts-0/wal/instance:
uuid: "af19c8a20ffb4b92841fd89231c562ba"
format_stamp: "Formatted at 2026-09-28 20:23:33 on dist-test-slave-b83v"
server_key: "6fd6ccf694a20943a12ebcef9f9d8cef"
server_key_iv: "a5131792c45b59a87fe54b9115f44b2b"
server_key_version: "encryptionkey@0"
I20260928 20:23:33.091413 22890 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:23:33.092310 22904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:33.092506 22890 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 20:23:33.092631 22890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790626764250411-3144-0/minicluster-data/ts-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790626764250411-3144-0/minicluster-data/ts-0/wal
uuid: "af19c8a20ffb4b92841fd89231c562ba"
format_stamp: "Formatted at 2026-09-28 20:23:33 on dist-test-slave-b83v"
server_key: "6fd6ccf694a20943a12ebcef9f9d8cef"
server_key_iv: "a5131792c45b59a87fe54b9115f44b2b"
server_key_version: "encryptionkey@0"
I20260928 20:23:33.092710 22890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790626764250411-3144-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790626764250411-3144-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790626764250411-3144-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:23:33.113467 22890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:23:33.113790 22890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:23:33.113945 22890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:23:33.114192 22890 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:23:33.114537 22890 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:23:33.114593 22890 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:33.114615 22890 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:23:33.114661 22890 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:33.120625 22890 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.1:43897
I20260928 20:23:33.120739 23017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.1:43897 every 8 connection(s)
I20260928 20:23:33.121054 22890 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790626764250411-3144-0/minicluster-data/ts-0/data/info.pb
I20260928 20:23:33.126556 22870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:33.121992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:41635 (local address 127.3.18.62:39971)
0928 20:23:33.122060 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:33.122063 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:33.122316 (+ 253us) server_negotiation.cc:407] Connection header received
0928 20:23:33.122789 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:33.122790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:33.122805 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:33.122879 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:33.123555 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:33.124096 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:33.125090 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:33.125285 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:33.125353 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:33.126254 (+ 901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:33.126255 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:33.126255 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:23:33.126260 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:33.126276 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:23:33.126280 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:33.126336 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:33.126453 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 20:23:33.126479 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 20:23:33.126914 23020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:33.121976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:39971 (local address 127.3.18.1:41635)
0928 20:23:33.122187 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:33.122231 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 20:23:33.122697 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:23:33.122944 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:23:33.122947 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:23:33.122955 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:23:33.123451 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:33.123456 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:33.124184 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:33.124185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:33.124981 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:33.124984 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:33.125372 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:33.125373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:33.125489 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:33.126084 (+ 595us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 20:23:33.126085 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:23:33.126116 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 20:23:33.126140 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:23:33.126361 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:23:33.126367 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:23:33.126370 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 20:23:33.126436 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 20:23:33.126521 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":78,"threads_started":1}
I20260928 20:23:33.127588 23018 heartbeater.cc:347] Connected to a master server at 127.3.18.62:39971
I20260928 20:23:33.127699 23018 heartbeater.cc:464] Registering TS with master...
I20260928 20:23:33.127925 23018 heartbeater.cc:511] Master 127.3.18.62:39971 requested a full tablet report, sending...
I20260928 20:23:33.128504 22830 ts_manager.cc:194] Registered new tserver with Master: af19c8a20ffb4b92841fd89231c562ba (127.3.18.1:43897)
I20260928 20:23:33.129176 22830 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.18.1:41635
I20260928 20:23:33.129201 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 22890
I20260928 20:23:33.129343 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790626764250411-3144-0/minicluster-data/ts-0/wal/instance
I20260928 20:23:33.129570 3144 external_mini_cluster.cc:1428] Setting key 45fce6dcbe8823698b0496c5b5b7a6c5
I20260928 20:23:33.130548 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790626764250411-3144-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790626764250411-3144-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790626764250411-3144-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.2:0
--local_ip_for_outbound_sockets=127.3.18.2
--webserver_interface=127.3.18.2
--webserver_port=0
--tserver_master_addrs=127.3.18.62:39971
--builtin_ntp_servers=127.3.18.20:33007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 20:23:33.215363 23021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:23:33.215727 23021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:23:33.215811 23021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:23:33.218062 23021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:23:33.218170 23021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:23:33.218214 23021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.2
I20260928 20:23:33.220918 23021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:33007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790626764250411-3144-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790626764250411-3144-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.18.2
--webserver_port=0
--tserver_master_addrs=127.3.18.62:39971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.2
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790626764250411-3144-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:23:33.221248 23021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:23:33.221650 23021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:23:33.227600 23029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:33.227603 23027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:33.227589 23026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:23:33.228013 23021 server_base.cc:1034] running on GCE node
I20260928 20:23:33.228292 23021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:23:33.228596 23021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:23:33.229842 23021 hybrid_clock.cc:648] HybridClock initialized: now 1790627013229815 us; error 40 us; skew 500 ppm
I20260928 20:23:33.231482 23021 webserver.cc:460] Webserver started at http://127.3.18.2:44143/ using document root <none> and password file <none>
I20260928 20:23:33.231928 23021 fs_manager.cc:359] Metadata directory not provided
I20260928 20:23:33.231992 23021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:23:33.232149 23021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:23:33.233355 23021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790626764250411-3144-0/minicluster-data/ts-1/data/instance:
uuid: "2ae7fc60171f49298aba96c606bb2286"
format_stamp: "Formatted at 2026-09-28 20:23:33 on dist-test-slave-b83v"
server_key: "c4a92ba4650f9d7aca863ea54d4f0da1"
server_key_iv: "00f7eda713885ff15c54f08dd723606f"
server_key_version: "encryptionkey@0"
I20260928 20:23:33.233789 23021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790626764250411-3144-0/minicluster-data/ts-1/wal/instance:
uuid: "2ae7fc60171f49298aba96c606bb2286"
format_stamp: "Formatted at 2026-09-28 20:23:33 on dist-test-slave-b83v"
server_key: "c4a92ba4650f9d7aca863ea54d4f0da1"
server_key_iv: "00f7eda713885ff15c54f08dd723606f"
server_key_version: "encryptionkey@0"
I20260928 20:23:33.235373 23021 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 20:23:33.236380 23035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:33.236610 23021 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 20:23:33.236706 23021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790626764250411-3144-0/minicluster-data/ts-1/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790626764250411-3144-0/minicluster-data/ts-1/wal
uuid: "2ae7fc60171f49298aba96c606bb2286"
format_stamp: "Formatted at 2026-09-28 20:23:33 on dist-test-slave-b83v"
server_key: "c4a92ba4650f9d7aca863ea54d4f0da1"
server_key_iv: "00f7eda713885ff15c54f08dd723606f"
server_key_version: "encryptionkey@0"
I20260928 20:23:33.236822 23021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790626764250411-3144-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790626764250411-3144-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790626764250411-3144-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:23:33.252350 23021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:23:33.252731 23021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:23:33.252912 23021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:23:33.253199 23021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:23:33.253559 23021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:23:33.253628 23021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:33.253672 23021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:23:33.253705 23021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:33.260186 23021 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.2:33469
I20260928 20:23:33.260251 23148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.2:33469 every 8 connection(s)
I20260928 20:23:33.260641 23021 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790626764250411-3144-0/minicluster-data/ts-1/data/info.pb
I20260928 20:23:33.266356 22870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:33.261516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.2:56525 (local address 127.3.18.62:39971)
0928 20:23:33.261593 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:33.261596 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:33.261866 (+ 270us) server_negotiation.cc:407] Connection header received
0928 20:23:33.262398 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:33.262400 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:33.262422 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:33.262484 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:33.263254 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:33.263859 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:33.265026 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:33.265220 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:33.265283 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:33.266129 (+ 846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:33.266130 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:33.266131 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:23:33.266136 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:33.266151 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:23:33.266154 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:33.266203 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:33.266272 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 20:23:33.266294 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 20:23:33.266677 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:33.261494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:39971 (local address 127.3.18.2:56525)
0928 20:23:33.261731 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:33.261768 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 20:23:33.262289 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:23:33.262565 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:23:33.262569 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:23:33.262584 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:23:33.263136 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:33.263140 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:33.263984 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:33.263985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:33.264932 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:33.264934 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:33.265300 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:33.265301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:33.265413 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:33.265989 (+ 576us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 20:23:33.265990 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:23:33.266019 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 20:23:33.266043 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:23:33.266206 (+ 163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:23:33.266209 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:23:33.266211 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 20:23:33.266261 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 20:23:33.266343 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":117,"threads_started":1}
I20260928 20:23:33.267263 23149 heartbeater.cc:347] Connected to a master server at 127.3.18.62:39971
I20260928 20:23:33.267362 23149 heartbeater.cc:464] Registering TS with master...
I20260928 20:23:33.267513 23149 heartbeater.cc:511] Master 127.3.18.62:39971 requested a full tablet report, sending...
I20260928 20:23:33.267938 22830 ts_manager.cc:194] Registered new tserver with Master: 2ae7fc60171f49298aba96c606bb2286 (127.3.18.2:33469)
I20260928 20:23:33.268420 22830 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.18.2:56525
I20260928 20:23:33.270457 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 23021
I20260928 20:23:33.270555 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790626764250411-3144-0/minicluster-data/ts-1/wal/instance
I20260928 20:23:33.270725 3144 external_mini_cluster.cc:1428] Setting key ee83018e4f25b750e0ac148f6765278b
I20260928 20:23:33.271724 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790626764250411-3144-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790626764250411-3144-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790626764250411-3144-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.18.3:0
--local_ip_for_outbound_sockets=127.3.18.3
--webserver_interface=127.3.18.3
--webserver_port=0
--tserver_master_addrs=127.3.18.62:39971
--builtin_ntp_servers=127.3.18.20:33007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 20:23:33.359953 23152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:23:33.360216 23152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:23:33.360263 23152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:23:33.361479 23152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:23:33.361552 23152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:23:33.361590 23152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.18.3
I20260928 20:23:33.362944 23152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:33007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790626764250411-3144-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790626764250411-3144-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.18.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.18.3
--webserver_port=0
--tserver_master_addrs=127.3.18.62:39971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.18.3
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790626764250411-3144-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:23:33.363224 23152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:23:33.363538 23152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:23:33.369071 23160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:33.369105 23158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:33.369105 23157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:23:33.369141 23152 server_base.cc:1034] running on GCE node
I20260928 20:23:33.369565 23152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:23:33.369831 23152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:23:33.371011 23152 hybrid_clock.cc:648] HybridClock initialized: now 1790627013370993 us; error 34 us; skew 500 ppm
I20260928 20:23:33.372665 23152 webserver.cc:460] Webserver started at http://127.3.18.3:33169/ using document root <none> and password file <none>
I20260928 20:23:33.372977 23152 fs_manager.cc:359] Metadata directory not provided
I20260928 20:23:33.373030 23152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:23:33.373173 23152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:23:33.374328 23152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790626764250411-3144-0/minicluster-data/ts-2/data/instance:
uuid: "d7176d64a44646739c4397e808f6cfd1"
format_stamp: "Formatted at 2026-09-28 20:23:33 on dist-test-slave-b83v"
server_key: "cb1300f5395fc8e158c05d00b8b531b0"
server_key_iv: "f1317a2fc332da64181829da7c21cc5b"
server_key_version: "encryptionkey@0"
I20260928 20:23:33.374902 23152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790626764250411-3144-0/minicluster-data/ts-2/wal/instance:
uuid: "d7176d64a44646739c4397e808f6cfd1"
format_stamp: "Formatted at 2026-09-28 20:23:33 on dist-test-slave-b83v"
server_key: "cb1300f5395fc8e158c05d00b8b531b0"
server_key_iv: "f1317a2fc332da64181829da7c21cc5b"
server_key_version: "encryptionkey@0"
I20260928 20:23:33.376396 23152 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:23:33.377511 23166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:33.377839 23152 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:23:33.377936 23152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790626764250411-3144-0/minicluster-data/ts-2/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790626764250411-3144-0/minicluster-data/ts-2/wal
uuid: "d7176d64a44646739c4397e808f6cfd1"
format_stamp: "Formatted at 2026-09-28 20:23:33 on dist-test-slave-b83v"
server_key: "cb1300f5395fc8e158c05d00b8b531b0"
server_key_iv: "f1317a2fc332da64181829da7c21cc5b"
server_key_version: "encryptionkey@0"
I20260928 20:23:33.377991 23152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790626764250411-3144-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790626764250411-3144-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790626764250411-3144-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:23:33.396840 23152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:23:33.397249 23152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:23:33.397375 23152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:23:33.397586 23152 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:23:33.398005 23152 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:23:33.398062 23152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:33.398094 23152 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:23:33.398121 23152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:33.404129 23152 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.3:37083
I20260928 20:23:33.404175 23279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.3:37083 every 8 connection(s)
I20260928 20:23:33.404533 23152 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790626764250411-3144-0/minicluster-data/ts-2/data/info.pb
I20260928 20:23:33.407122 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 23152
I20260928 20:23:33.407212 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790626764250411-3144-0/minicluster-data/ts-2/wal/instance
I20260928 20:23:33.407409 3144 external_mini_cluster.cc:1428] Setting key e1392adf1375e2cb72ea772a929f1b9a
I20260928 20:23:33.410319 22870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:33.405310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:49825 (local address 127.3.18.62:39971)
0928 20:23:33.405376 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:33.405378 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:33.405612 (+ 234us) server_negotiation.cc:407] Connection header received
0928 20:23:33.406096 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:33.406097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:33.406109 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:33.406167 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:33.406850 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:33.407421 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:33.408884 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:33.409075 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:33.409139 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:33.410080 (+ 941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:33.410082 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:33.410082 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:23:33.410087 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:33.410101 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:23:33.410104 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:33.410154 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:33.410227 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 20:23:33.410255 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 20:23:33.410655 23282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:33.405270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:39971 (local address 127.3.18.3:49825)
0928 20:23:33.405473 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:33.405509 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 20:23:33.405994 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:23:33.406227 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:23:33.406230 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:23:33.406239 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:23:33.406763 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:33.406767 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:33.407556 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:33.407557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:33.408770 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:33.408773 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:33.409177 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:33.409179 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:33.409313 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:33.409899 (+ 586us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 20:23:33.409900 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:23:33.409934 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 20:23:33.409959 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:23:33.410168 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:23:33.410173 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:23:33.410175 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 20:23:33.410229 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 20:23:33.410315 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":74,"threads_started":1}
I20260928 20:23:33.411229 23280 heartbeater.cc:347] Connected to a master server at 127.3.18.62:39971
I20260928 20:23:33.411324 23280 heartbeater.cc:464] Registering TS with master...
I20260928 20:23:33.411527 23280 heartbeater.cc:511] Master 127.3.18.62:39971 requested a full tablet report, sending...
I20260928 20:23:33.412127 22830 ts_manager.cc:194] Registered new tserver with Master: d7176d64a44646739c4397e808f6cfd1 (127.3.18.3:37083)
I20260928 20:23:33.412514 22830 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.18.3:49825
I20260928 20:23:33.419435 3144 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 20:23:33.424180 22870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:33.421034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36016 (local address 127.3.18.62:39971)
0928 20:23:33.421121 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:33.421124 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:33.421247 (+ 123us) server_negotiation.cc:407] Connection header received
0928 20:23:33.421340 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:33.421340 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:33.421352 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:33.421397 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:33.421590 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:33.422106 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:33.422889 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:33.423099 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:33.423174 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:33.423919 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:33.423920 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:33.423921 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:23:33.423925 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:33.423940 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:23:33.423944 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:33.424013 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:33.424065 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 20:23:33.424091 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 20:23:33.425987 22830 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36016:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 20:23:33.427513 22830 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 20:23:33.434438 23293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:33.430175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49962 (local address 127.3.18.3:37083)
0928 20:23:33.430710 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:33.430714 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:33.430723 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:23:33.430761 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:33.430763 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:33.430779 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:33.430850 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:23:33.431032 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:33.431847 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:33.433184 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:33.433935 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:33.434018 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:33.434120 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:33.434296 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 20:23:33.434339 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":72,"threads_started":1}
I20260928 20:23:33.434448 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:33.429895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.1:43897 (local address 127.0.0.1:52594)
0928 20:23:33.430080 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:33.430123 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 20:23:33.430191 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:23:33.430545 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:23:33.430548 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:23:33.430550 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:23:33.430629 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:33.430632 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:33.432049 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:33.432049 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:33.433284 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:33.433287 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:33.434172 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:33.434173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:33.434242 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:33.434245 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 20:23:33.434297 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 20:23:33.434334 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":54,"thread_start_us":66,"threads_started":1}
I20260928 20:23:33.434448 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:33.429903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.3:37083 (local address 127.0.0.1:49962)
0928 20:23:33.430408 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:33.430415 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 20:23:33.430441 (+ 26us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:23:33.430923 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:23:33.430924 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:23:33.430925 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:23:33.430961 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:33.430962 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:33.431939 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:33.431941 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:33.433074 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:33.433077 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:33.434067 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:33.434068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:33.434168 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:33.434185 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 20:23:33.434272 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 20:23:33.434334 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"mutex_wait_us":82,"thread_start_us":59,"threads_started":1}
I20260928 20:23:33.435014 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:33.430069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52594 (local address 127.3.18.1:43897)
0928 20:23:33.430350 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:33.430354 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:33.430363 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:23:33.430403 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:33.430405 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:33.430421 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:33.430517 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:23:33.430728 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:33.431963 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:33.433374 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:33.434079 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:33.434145 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:33.434225 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:33.434888 (+ 663us) server_negotiation.cc:299] Negotiation successful
0928 20:23:33.434935 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":95,"threads_started":1}
I20260928 20:23:33.435438 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:33.429898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.2:33469 (local address 127.0.0.1:45190)
0928 20:23:33.430268 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:33.430279 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:23:33.430341 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:23:33.430545 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:23:33.430548 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:23:33.430553 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:23:33.430629 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:33.430631 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:33.431981 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:33.431982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:33.433288 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:33.433290 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:33.435203 (+ 1913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:33.435204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:33.435267 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:33.435271 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:23:33.435332 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 20:23:33.435370 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"mutex_wait_us":54,"thread_start_us":72,"threads_started":1}
I20260928 20:23:33.435482 23292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:33.430100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45190 (local address 127.3.18.2:33469)
0928 20:23:33.430282 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:33.430287 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:33.430337 (+ 50us) server_negotiation.cc:407] Connection header received
0928 20:23:33.430405 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:33.430407 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:33.430421 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:33.430518 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:23:33.430715 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:33.431883 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:33.434280 (+ 2397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:33.435069 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:33.435133 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:33.435225 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:33.435357 (+ 132us) server_negotiation.cc:299] Negotiation successful
0928 20:23:33.435398 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":82,"threads_started":1}
I20260928 20:23:33.435511 23214 tablet_service.cc:1467] Processing CreateTablet for tablet 59d8302bb4304f698ceb14eb3791d7ce (DEFAULT_TABLE table=test-table [id=74cfbc96199b416dbd5b3a011fa3833d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:23:33.435802 22952 tablet_service.cc:1467] Processing CreateTablet for tablet 59d8302bb4304f698ceb14eb3791d7ce (DEFAULT_TABLE table=test-table [id=74cfbc96199b416dbd5b3a011fa3833d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:23:33.435937 23214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59d8302bb4304f698ceb14eb3791d7ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:23:33.436177 22952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59d8302bb4304f698ceb14eb3791d7ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:23:33.436395 23083 tablet_service.cc:1467] Processing CreateTablet for tablet 59d8302bb4304f698ceb14eb3791d7ce (DEFAULT_TABLE table=test-table [id=74cfbc96199b416dbd5b3a011fa3833d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:23:33.436698 23083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59d8302bb4304f698ceb14eb3791d7ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:23:33.438968 23294 tablet_bootstrap.cc:492] T 59d8302bb4304f698ceb14eb3791d7ce P d7176d64a44646739c4397e808f6cfd1: Bootstrap starting.
I20260928 20:23:33.439548 23294 tablet_bootstrap.cc:654] T 59d8302bb4304f698ceb14eb3791d7ce P d7176d64a44646739c4397e808f6cfd1: Neither blocks nor log segments found. Creating new log.
I20260928 20:23:33.439932 23294 log.cc:824] T 59d8302bb4304f698ceb14eb3791d7ce P d7176d64a44646739c4397e808f6cfd1: Log is configured to *not* fsync() on all Append() calls
I20260928 20:23:33.439914 23296 tablet_bootstrap.cc:492] T 59d8302bb4304f698ceb14eb3791d7ce P 2ae7fc60171f49298aba96c606bb2286: Bootstrap starting.
I20260928 20:23:33.440201 23295 tablet_bootstrap.cc:492] T 59d8302bb4304f698ceb14eb3791d7ce P af19c8a20ffb4b92841fd89231c562ba: Bootstrap starting.
I20260928 20:23:33.440656 23294 tablet_bootstrap.cc:492] T 59d8302bb4304f698ceb14eb3791d7ce P d7176d64a44646739c4397e808f6cfd1: No bootstrap required, opened a new log
I20260928 20:23:33.440660 23296 tablet_bootstrap.cc:654] T 59d8302bb4304f698ceb14eb3791d7ce P 2ae7fc60171f49298aba96c606bb2286: Neither blocks nor log segments found. Creating new log.
I20260928 20:23:33.440737 23294 ts_tablet_manager.cc:1397] T 59d8302bb4304f698ceb14eb3791d7ce P d7176d64a44646739c4397e808f6cfd1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 20:23:33.440887 23295 tablet_bootstrap.cc:654] T 59d8302bb4304f698ceb14eb3791d7ce P af19c8a20ffb4b92841fd89231c562ba: Neither blocks nor log segments found. Creating new log.
I20260928 20:23:33.440963 23296 log.cc:824] T 59d8302bb4304f698ceb14eb3791d7ce P 2ae7fc60171f49298aba96c606bb2286: Log is configured to *not* fsync() on all Append() calls
I20260928 20:23:33.441293 23295 log.cc:824] T 59d8302bb4304f698ceb14eb3791d7ce P af19c8a20ffb4b92841fd89231c562ba: Log is configured to *not* fsync() on all Append() calls
I20260928 20:23:33.441696 23296 tablet_bootstrap.cc:492] T 59d8302bb4304f698ceb14eb3791d7ce P 2ae7fc60171f49298aba96c606bb2286: No bootstrap required, opened a new log
I20260928 20:23:33.441761 23296 ts_tablet_manager.cc:1397] T 59d8302bb4304f698ceb14eb3791d7ce P 2ae7fc60171f49298aba96c606bb2286: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 20:23:33.442009 23295 tablet_bootstrap.cc:492] T 59d8302bb4304f698ceb14eb3791d7ce P af19c8a20ffb4b92841fd89231c562ba: No bootstrap required, opened a new log
I20260928 20:23:33.442075 23295 ts_tablet_manager.cc:1397] T 59d8302bb4304f698ceb14eb3791d7ce P af19c8a20ffb4b92841fd89231c562ba: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 20:23:33.443349 23294 raft_consensus.cc:348] T 59d8302bb4304f698ceb14eb3791d7ce P d7176d64a44646739c4397e808f6cfd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af19c8a20ffb4b92841fd89231c562ba" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 43897 } } peers { permanent_uuid: "2ae7fc60171f49298aba96c606bb2286" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 33469 } } peers { permanent_uuid: "d7176d64a44646739c4397e808f6cfd1" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 37083 } }
I20260928 20:23:33.443527 23294 raft_consensus.cc:374] T 59d8302bb4304f698ceb14eb3791d7ce P d7176d64a44646739c4397e808f6cfd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:23:33.443567 23294 raft_consensus.cc:729] T 59d8302bb4304f698ceb14eb3791d7ce P d7176d64a44646739c4397e808f6cfd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7176d64a44646739c4397e808f6cfd1, State: Initialized, Role: FOLLOWER
I20260928 20:23:33.443697 23294 consensus_queue.cc:260] T 59d8302bb4304f698ceb14eb3791d7ce P d7176d64a44646739c4397e808f6cfd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af19c8a20ffb4b92841fd89231c562ba" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 43897 } } peers { permanent_uuid: "2ae7fc60171f49298aba96c606bb2286" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 33469 } } peers { permanent_uuid: "d7176d64a44646739c4397e808f6cfd1" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 37083 } }
I20260928 20:23:33.443763 23296 raft_consensus.cc:348] T 59d8302bb4304f698ceb14eb3791d7ce P 2ae7fc60171f49298aba96c606bb2286 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af19c8a20ffb4b92841fd89231c562ba" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 43897 } } peers { permanent_uuid: "2ae7fc60171f49298aba96c606bb2286" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 33469 } } peers { permanent_uuid: "d7176d64a44646739c4397e808f6cfd1" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 37083 } }
I20260928 20:23:33.443962 23296 raft_consensus.cc:374] T 59d8302bb4304f698ceb14eb3791d7ce P 2ae7fc60171f49298aba96c606bb2286 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:23:33.444006 23296 raft_consensus.cc:729] T 59d8302bb4304f698ceb14eb3791d7ce P 2ae7fc60171f49298aba96c606bb2286 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ae7fc60171f49298aba96c606bb2286, State: Initialized, Role: FOLLOWER
I20260928 20:23:33.443995 23294 ts_tablet_manager.cc:1428] T 59d8302bb4304f698ceb14eb3791d7ce P d7176d64a44646739c4397e808f6cfd1: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260928 20:23:33.444113 23296 consensus_queue.cc:260] T 59d8302bb4304f698ceb14eb3791d7ce P 2ae7fc60171f49298aba96c606bb2286 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af19c8a20ffb4b92841fd89231c562ba" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 43897 } } peers { permanent_uuid: "2ae7fc60171f49298aba96c606bb2286" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 33469 } } peers { permanent_uuid: "d7176d64a44646739c4397e808f6cfd1" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 37083 } }
I20260928 20:23:33.444372 23296 ts_tablet_manager.cc:1428] T 59d8302bb4304f698ceb14eb3791d7ce P 2ae7fc60171f49298aba96c606bb2286: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 20:23:33.444433 23149 heartbeater.cc:503] Master 127.3.18.62:39971 was elected leader, sending a full tablet report...
I20260928 20:23:33.444689 23295 raft_consensus.cc:348] T 59d8302bb4304f698ceb14eb3791d7ce P af19c8a20ffb4b92841fd89231c562ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af19c8a20ffb4b92841fd89231c562ba" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 43897 } } peers { permanent_uuid: "2ae7fc60171f49298aba96c606bb2286" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 33469 } } peers { permanent_uuid: "d7176d64a44646739c4397e808f6cfd1" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 37083 } }
I20260928 20:23:33.444821 23295 raft_consensus.cc:374] T 59d8302bb4304f698ceb14eb3791d7ce P af19c8a20ffb4b92841fd89231c562ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:23:33.444882 23295 raft_consensus.cc:729] T 59d8302bb4304f698ceb14eb3791d7ce P af19c8a20ffb4b92841fd89231c562ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af19c8a20ffb4b92841fd89231c562ba, State: Initialized, Role: FOLLOWER
I20260928 20:23:33.444988 23295 consensus_queue.cc:260] T 59d8302bb4304f698ceb14eb3791d7ce P af19c8a20ffb4b92841fd89231c562ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af19c8a20ffb4b92841fd89231c562ba" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 43897 } } peers { permanent_uuid: "2ae7fc60171f49298aba96c606bb2286" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 33469 } } peers { permanent_uuid: "d7176d64a44646739c4397e808f6cfd1" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 37083 } }
I20260928 20:23:33.445220 23295 ts_tablet_manager.cc:1428] T 59d8302bb4304f698ceb14eb3791d7ce P af19c8a20ffb4b92841fd89231c562ba: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 20:23:33.445271 23018 heartbeater.cc:503] Master 127.3.18.62:39971 was elected leader, sending a full tablet report...
I20260928 20:23:33.445369 23280 heartbeater.cc:503] Master 127.3.18.62:39971 was elected leader, sending a full tablet report...
W20260928 20:23:33.511289 23150 tablet.cc:2369] T 59d8302bb4304f698ceb14eb3791d7ce P 2ae7fc60171f49298aba96c606bb2286: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 20:23:33.544767 23302 raft_consensus.cc:482] T 59d8302bb4304f698ceb14eb3791d7ce P af19c8a20ffb4b92841fd89231c562ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:23:33.544942 23302 raft_consensus.cc:504] T 59d8302bb4304f698ceb14eb3791d7ce P af19c8a20ffb4b92841fd89231c562ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af19c8a20ffb4b92841fd89231c562ba" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 43897 } } peers { permanent_uuid: "2ae7fc60171f49298aba96c606bb2286" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 33469 } } peers { permanent_uuid: "d7176d64a44646739c4397e808f6cfd1" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 37083 } }
I20260928 20:23:33.545035 23300 raft_consensus.cc:482] T 59d8302bb4304f698ceb14eb3791d7ce P d7176d64a44646739c4397e808f6cfd1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:23:33.545200 23300 raft_consensus.cc:504] T 59d8302bb4304f698ceb14eb3791d7ce P d7176d64a44646739c4397e808f6cfd1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af19c8a20ffb4b92841fd89231c562ba" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 43897 } } peers { permanent_uuid: "2ae7fc60171f49298aba96c606bb2286" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 33469 } } peers { permanent_uuid: "d7176d64a44646739c4397e808f6cfd1" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 37083 } }
I20260928 20:23:33.545480 23302 leader_election.cc:288] T 59d8302bb4304f698ceb14eb3791d7ce P af19c8a20ffb4b92841fd89231c562ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ae7fc60171f49298aba96c606bb2286 (127.3.18.2:33469), d7176d64a44646739c4397e808f6cfd1 (127.3.18.3:37083)
I20260928 20:23:33.545608 23300 leader_election.cc:288] T 59d8302bb4304f698ceb14eb3791d7ce P d7176d64a44646739c4397e808f6cfd1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af19c8a20ffb4b92841fd89231c562ba (127.3.18.1:43897), 2ae7fc60171f49298aba96c606bb2286 (127.3.18.2:33469)
I20260928 20:23:33.549930 23020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:33.545674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.2:33469 (local address 127.3.18.1:35429)
0928 20:23:33.546306 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:33.546320 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 20:23:33.546376 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:23:33.546768 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:23:33.546768 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:23:33.546770 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:23:33.546846 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:33.546848 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:33.547673 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:33.547675 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:33.548877 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:33.548880 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:33.549657 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:33.549658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:33.549728 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:33.549732 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:23:33.549782 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 20:23:33.549842 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"mutex_wait_us":1}
I20260928 20:23:33.549995 23293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:33.545762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:39097 (local address 127.3.18.3:37083)
0928 20:23:33.545969 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:33.545971 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:33.545983 (+ 12us) server_negotiation.cc:407] Connection header received
0928 20:23:33.546094 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:33.546095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:33.546105 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:33.546140 (+ 35us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:23:33.546407 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:33.547263 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:33.549026 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:33.549560 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:33.549621 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:33.549632 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:33.549882 (+ 250us) server_negotiation.cc:299] Negotiation successful
0928 20:23:33.549923 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184}
I20260928 20:23:33.550275 23303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:33.545697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.3:37083 (local address 127.3.18.1:39097)
0928 20:23:33.545902 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:33.545913 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:23:33.545978 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:23:33.546228 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:23:33.546229 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:23:33.546232 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:23:33.546314 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:33.546318 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:33.547360 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:33.547361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:33.548914 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:33.548918 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:33.549700 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:33.549701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:33.549786 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:33.549796 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 20:23:33.550123 (+ 327us) client_negotiation.cc:240] Negotiation successful
0928 20:23:33.550185 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":43,"thread_start_us":103,"threads_started":1}
I20260928 20:23:33.550420 23292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:33.546039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.1:35429 (local address 127.3.18.2:33469)
0928 20:23:33.546100 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:33.546102 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:33.546619 (+ 517us) server_negotiation.cc:407] Connection header received
0928 20:23:33.546653 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:33.546654 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:33.546665 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:33.546718 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:23:33.546926 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:33.547582 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:33.548989 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:33.549538 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:33.549603 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:33.549613 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:33.550295 (+ 682us) server_negotiation.cc:299] Negotiation successful
0928 20:23:33.550338 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 20:23:33.550632 23234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59d8302bb4304f698ceb14eb3791d7ce" candidate_uuid: "af19c8a20ffb4b92841fd89231c562ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7176d64a44646739c4397e808f6cfd1" is_pre_election: true
I20260928 20:23:33.550745 23282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:33.545712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.1:43897 (local address 127.3.18.3:46259)
0928 20:23:33.545860 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:33.545874 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 20:23:33.545923 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:23:33.547648 (+ 1725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:23:33.547649 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:23:33.547651 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:23:33.547718 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:33.547719 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:33.548645 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:33.548646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:33.549746 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:33.549749 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:33.550473 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:33.550474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:33.550556 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:33.550559 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 20:23:33.550625 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 20:23:33.550668 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131}
I20260928 20:23:33.550699 23103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59d8302bb4304f698ceb14eb3791d7ce" candidate_uuid: "af19c8a20ffb4b92841fd89231c562ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae7fc60171f49298aba96c606bb2286" is_pre_election: true
I20260928 20:23:33.550782 23234 raft_consensus.cc:2454] T 59d8302bb4304f698ceb14eb3791d7ce P d7176d64a44646739c4397e808f6cfd1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af19c8a20ffb4b92841fd89231c562ba in term 0.
I20260928 20:23:33.550863 23103 raft_consensus.cc:2454] T 59d8302bb4304f698ceb14eb3791d7ce P 2ae7fc60171f49298aba96c606bb2286 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af19c8a20ffb4b92841fd89231c562ba in term 0.
I20260928 20:23:33.551033 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:33.547330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:46259 (local address 127.3.18.1:43897)
0928 20:23:33.547368 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:33.547369 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:33.547376 (+ 7us) server_negotiation.cc:407] Connection header received
0928 20:23:33.547395 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:33.547395 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:33.547404 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:33.547453 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:23:33.547820 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:33.548530 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:33.549835 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:33.550367 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:33.550427 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:33.550436 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:33.550911 (+ 475us) server_negotiation.cc:299] Negotiation successful
0928 20:23:33.550960 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 20:23:33.551057 22908 leader_election.cc:302] T 59d8302bb4304f698ceb14eb3791d7ce P af19c8a20ffb4b92841fd89231c562ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af19c8a20ffb4b92841fd89231c562ba, d7176d64a44646739c4397e808f6cfd1; no voters:
I20260928 20:23:33.551230 23302 raft_consensus.cc:2781] T 59d8302bb4304f698ceb14eb3791d7ce P af19c8a20ffb4b92841fd89231c562ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 20:23:33.551309 23302 raft_consensus.cc:482] T 59d8302bb4304f698ceb14eb3791d7ce P af19c8a20ffb4b92841fd89231c562ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 20:23:33.551337 23302 raft_consensus.cc:3037] T 59d8302bb4304f698ceb14eb3791d7ce P af19c8a20ffb4b92841fd89231c562ba [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:23:33.551514 23304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:33.546415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.2:33469 (local address 127.3.18.3:47605)
0928 20:23:33.546639 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:33.546650 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:23:33.546724 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:23:33.547124 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:23:33.547125 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:23:33.547127 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:23:33.547193 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:33.547195 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:33.548285 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:33.548286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:33.549662 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:33.549664 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:33.551051 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:33.551053 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:33.551220 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:33.551226 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 20:23:33.551296 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 20:23:33.551423 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":117,"threads_started":1}
I20260928 20:23:33.551467 22972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59d8302bb4304f698ceb14eb3791d7ce" candidate_uuid: "d7176d64a44646739c4397e808f6cfd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af19c8a20ffb4b92841fd89231c562ba" is_pre_election: true
I20260928 20:23:33.551806 23305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:33.546486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.18.3:47605 (local address 127.3.18.2:33469)
0928 20:23:33.547007 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:33.547008 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:33.547016 (+ 8us) server_negotiation.cc:407] Connection header received
0928 20:23:33.547033 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:33.547034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:33.547042 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:33.547106 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:23:33.547286 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:33.548149 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:33.549758 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:33.550888 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:33.551012 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:33.551025 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:33.551663 (+ 638us) server_negotiation.cc:299] Negotiation successful
0928 20:23:33.551713 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":108,"threads_started":1}
I20260928 20:23:33.552022 23103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59d8302bb4304f698ceb14eb3791d7ce" candidate_uuid: "d7176d64a44646739c4397e808f6cfd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae7fc60171f49298aba96c606bb2286" is_pre_election: true
I20260928 20:23:33.552119 23103 raft_consensus.cc:2454] T 59d8302bb4304f698ceb14eb3791d7ce P 2ae7fc60171f49298aba96c606bb2286 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7176d64a44646739c4397e808f6cfd1 in term 0.
I20260928 20:23:33.552285 23169 leader_election.cc:302] T 59d8302bb4304f698ceb14eb3791d7ce P d7176d64a44646739c4397e808f6cfd1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ae7fc60171f49298aba96c606bb2286, d7176d64a44646739c4397e808f6cfd1; no voters:
I20260928 20:23:33.552443 23300 raft_consensus.cc:2781] T 59d8302bb4304f698ceb14eb3791d7ce P d7176d64a44646739c4397e808f6cfd1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 20:23:33.552424 23302 raft_consensus.cc:504] T 59d8302bb4304f698ceb14eb3791d7ce P af19c8a20ffb4b92841fd89231c562ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af19c8a20ffb4b92841fd89231c562ba" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 43897 } } peers { permanent_uuid: "2ae7fc60171f49298aba96c606bb2286" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 33469 } } peers { permanent_uuid: "d7176d64a44646739c4397e808f6cfd1" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 37083 } }
I20260928 20:23:33.552516 23300 raft_consensus.cc:482] T 59d8302bb4304f698ceb14eb3791d7ce P d7176d64a44646739c4397e808f6cfd1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 20:23:33.552547 23300 raft_consensus.cc:3037] T 59d8302bb4304f698ceb14eb3791d7ce P d7176d64a44646739c4397e808f6cfd1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:23:33.552628 22972 raft_consensus.cc:2379] T 59d8302bb4304f698ceb14eb3791d7ce P af19c8a20ffb4b92841fd89231c562ba [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7176d64a44646739c4397e808f6cfd1 in current term 1: Already voted for candidate af19c8a20ffb4b92841fd89231c562ba in this term.
I20260928 20:23:33.552749 23103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59d8302bb4304f698ceb14eb3791d7ce" candidate_uuid: "af19c8a20ffb4b92841fd89231c562ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae7fc60171f49298aba96c606bb2286"
I20260928 20:23:33.552772 23234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59d8302bb4304f698ceb14eb3791d7ce" candidate_uuid: "af19c8a20ffb4b92841fd89231c562ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7176d64a44646739c4397e808f6cfd1"
I20260928 20:23:33.552815 23103 raft_consensus.cc:3037] T 59d8302bb4304f698ceb14eb3791d7ce P 2ae7fc60171f49298aba96c606bb2286 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:23:33.552932 23302 leader_election.cc:288] T 59d8302bb4304f698ceb14eb3791d7ce P af19c8a20ffb4b92841fd89231c562ba [CANDIDATE]: Term 1 election: Requested vote from peers 2ae7fc60171f49298aba96c606bb2286 (127.3.18.2:33469), d7176d64a44646739c4397e808f6cfd1 (127.3.18.3:37083)
I20260928 20:23:33.553445 23300 raft_consensus.cc:504] T 59d8302bb4304f698ceb14eb3791d7ce P d7176d64a44646739c4397e808f6cfd1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af19c8a20ffb4b92841fd89231c562ba" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 43897 } } peers { permanent_uuid: "2ae7fc60171f49298aba96c606bb2286" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 33469 } } peers { permanent_uuid: "d7176d64a44646739c4397e808f6cfd1" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 37083 } }
I20260928 20:23:33.553633 23103 raft_consensus.cc:2454] T 59d8302bb4304f698ceb14eb3791d7ce P 2ae7fc60171f49298aba96c606bb2286 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af19c8a20ffb4b92841fd89231c562ba in term 1.
I20260928 20:23:33.553640 23300 leader_election.cc:288] T 59d8302bb4304f698ceb14eb3791d7ce P d7176d64a44646739c4397e808f6cfd1 [CANDIDATE]: Term 1 election: Requested vote from peers af19c8a20ffb4b92841fd89231c562ba (127.3.18.1:43897), 2ae7fc60171f49298aba96c606bb2286 (127.3.18.2:33469)
I20260928 20:23:33.553655 23234 raft_consensus.cc:2379] T 59d8302bb4304f698ceb14eb3791d7ce P d7176d64a44646739c4397e808f6cfd1 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate af19c8a20ffb4b92841fd89231c562ba in current term 1: Already voted for candidate d7176d64a44646739c4397e808f6cfd1 in this term.
I20260928 20:23:33.553840 22907 leader_election.cc:302] T 59d8302bb4304f698ceb14eb3791d7ce P af19c8a20ffb4b92841fd89231c562ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ae7fc60171f49298aba96c606bb2286, af19c8a20ffb4b92841fd89231c562ba; no voters:
I20260928 20:23:33.553843 23103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59d8302bb4304f698ceb14eb3791d7ce" candidate_uuid: "d7176d64a44646739c4397e808f6cfd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae7fc60171f49298aba96c606bb2286"
I20260928 20:23:33.553925 23103 raft_consensus.cc:2379] T 59d8302bb4304f698ceb14eb3791d7ce P 2ae7fc60171f49298aba96c606bb2286 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d7176d64a44646739c4397e808f6cfd1 in current term 1: Already voted for candidate af19c8a20ffb4b92841fd89231c562ba in this term.
I20260928 20:23:33.553977 22972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59d8302bb4304f698ceb14eb3791d7ce" candidate_uuid: "d7176d64a44646739c4397e808f6cfd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af19c8a20ffb4b92841fd89231c562ba"
I20260928 20:23:33.554039 22972 raft_consensus.cc:2379] T 59d8302bb4304f698ceb14eb3791d7ce P af19c8a20ffb4b92841fd89231c562ba [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d7176d64a44646739c4397e808f6cfd1 in current term 1: Already voted for candidate af19c8a20ffb4b92841fd89231c562ba in this term.
I20260928 20:23:33.554132 23302 raft_consensus.cc:2781] T 59d8302bb4304f698ceb14eb3791d7ce P af19c8a20ffb4b92841fd89231c562ba [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:23:33.554198 23167 leader_election.cc:302] T 59d8302bb4304f698ceb14eb3791d7ce P d7176d64a44646739c4397e808f6cfd1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7176d64a44646739c4397e808f6cfd1; no voters: 2ae7fc60171f49298aba96c606bb2286, af19c8a20ffb4b92841fd89231c562ba
I20260928 20:23:33.554289 23302 raft_consensus.cc:686] T 59d8302bb4304f698ceb14eb3791d7ce P af19c8a20ffb4b92841fd89231c562ba [term 1 LEADER]: Becoming Leader. State: Replica: af19c8a20ffb4b92841fd89231c562ba, State: Running, Role: LEADER
I20260928 20:23:33.554296 23300 raft_consensus.cc:2726] T 59d8302bb4304f698ceb14eb3791d7ce P d7176d64a44646739c4397e808f6cfd1 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260928 20:23:33.554473 23302 consensus_queue.cc:237] T 59d8302bb4304f698ceb14eb3791d7ce P af19c8a20ffb4b92841fd89231c562ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af19c8a20ffb4b92841fd89231c562ba" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 43897 } } peers { permanent_uuid: "2ae7fc60171f49298aba96c606bb2286" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 33469 } } peers { permanent_uuid: "d7176d64a44646739c4397e808f6cfd1" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 37083 } }
I20260928 20:23:33.555374 22830 catalog_manager.cc:5495] T 59d8302bb4304f698ceb14eb3791d7ce P af19c8a20ffb4b92841fd89231c562ba reported cstate change: term changed from 0 to 1, leader changed from <none> to af19c8a20ffb4b92841fd89231c562ba (127.3.18.1). New cstate: current_term: 1 leader_uuid: "af19c8a20ffb4b92841fd89231c562ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af19c8a20ffb4b92841fd89231c562ba" member_type: VOTER last_known_addr { host: "127.3.18.1" port: 43897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ae7fc60171f49298aba96c606bb2286" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 33469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7176d64a44646739c4397e808f6cfd1" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 37083 } health_report { overall_health: UNKNOWN } } }
I20260928 20:23:33.582033 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:33.579243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52598 (local address 127.3.18.1:43897)
0928 20:23:33.579310 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:33.579311 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:33.579340 (+ 29us) server_negotiation.cc:407] Connection header received
0928 20:23:33.579378 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:33.579379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:33.579398 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:33.579453 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:23:33.579600 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:33.580169 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:33.581143 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:33.581338 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:33.581397 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:33.581604 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:23:33.581729 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:23:33.581797 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:33.581877 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 20:23:33.581898 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 20:23:33.584178 23234 raft_consensus.cc:1261] T 59d8302bb4304f698ceb14eb3791d7ce P d7176d64a44646739c4397e808f6cfd1 [term 1 FOLLOWER]: Refusing update from remote peer af19c8a20ffb4b92841fd89231c562ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:23:33.584275 23103 raft_consensus.cc:1261] T 59d8302bb4304f698ceb14eb3791d7ce P 2ae7fc60171f49298aba96c606bb2286 [term 1 FOLLOWER]: Refusing update from remote peer af19c8a20ffb4b92841fd89231c562ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:23:33.584473 23302 consensus_queue.cc:1035] T 59d8302bb4304f698ceb14eb3791d7ce P af19c8a20ffb4b92841fd89231c562ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7176d64a44646739c4397e808f6cfd1" member_type: VOTER last_known_addr { host: "127.3.18.3" port: 37083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:23:33.584626 23306 consensus_queue.cc:1035] T 59d8302bb4304f698ceb14eb3791d7ce P af19c8a20ffb4b92841fd89231c562ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ae7fc60171f49298aba96c606bb2286" member_type: VOTER last_known_addr { host: "127.3.18.2" port: 33469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:23:33.587450 23309 mvcc.cc:205] Tried to move back new op lower bound from 7334408247637450752 to 7334408247519834112. Current Snapshot: MvccSnapshot[applied={T|T < 7334408247637450752}]
I20260928 20:23:33.588335 23311 mvcc.cc:205] Tried to move back new op lower bound from 7334408247637450752 to 7334408247519834112. Current Snapshot: MvccSnapshot[applied={T|T < 7334408247637450752}]
I20260928 20:23:33.589741 22830 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:23:33.589828 22830 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:23:33.590700 22830 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b5d22b2c396d438880b46f12d5580558: Sending DeleteTablet for 3 replicas of tablet 59d8302bb4304f698ceb14eb3791d7ce
I20260928 20:23:33.591077 23214 tablet_service.cc:1514] Processing DeleteTablet for tablet 59d8302bb4304f698ceb14eb3791d7ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:23:33 UTC) from {username='slave'} at 127.0.0.1:49962
I20260928 20:23:33.591092 23083 tablet_service.cc:1514] Processing DeleteTablet for tablet 59d8302bb4304f698ceb14eb3791d7ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:23:33 UTC) from {username='slave'} at 127.0.0.1:45190
I20260928 20:23:33.591506 22952 tablet_service.cc:1514] Processing DeleteTablet for tablet 59d8302bb4304f698ceb14eb3791d7ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:23:33 UTC) from {username='slave'} at 127.0.0.1:52594
I20260928 20:23:33.591801 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 22890
I20260928 20:23:33.592568 23318 tablet_replica.cc:330] T 59d8302bb4304f698ceb14eb3791d7ce P 2ae7fc60171f49298aba96c606bb2286: stopping tablet replica
I20260928 20:23:33.592732 23318 raft_consensus.cc:2229] T 59d8302bb4304f698ceb14eb3791d7ce P 2ae7fc60171f49298aba96c606bb2286 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:23:33.592891 23318 raft_consensus.cc:2258] T 59d8302bb4304f698ceb14eb3791d7ce P 2ae7fc60171f49298aba96c606bb2286 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:23:33.593103 23319 tablet_replica.cc:330] T 59d8302bb4304f698ceb14eb3791d7ce P d7176d64a44646739c4397e808f6cfd1: stopping tablet replica
I20260928 20:23:33.593223 23319 raft_consensus.cc:2229] T 59d8302bb4304f698ceb14eb3791d7ce P d7176d64a44646739c4397e808f6cfd1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:23:33.593339 23319 raft_consensus.cc:2258] T 59d8302bb4304f698ceb14eb3791d7ce P d7176d64a44646739c4397e808f6cfd1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:23:33.593684 23318 ts_tablet_manager.cc:1905] T 59d8302bb4304f698ceb14eb3791d7ce P 2ae7fc60171f49298aba96c606bb2286: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:23:33.593982 23319 ts_tablet_manager.cc:1905] T 59d8302bb4304f698ceb14eb3791d7ce P d7176d64a44646739c4397e808f6cfd1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:23:33.595947 23319 ts_tablet_manager.cc:1918] T 59d8302bb4304f698ceb14eb3791d7ce P d7176d64a44646739c4397e808f6cfd1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:23:33.595943 23318 ts_tablet_manager.cc:1918] T 59d8302bb4304f698ceb14eb3791d7ce P 2ae7fc60171f49298aba96c606bb2286: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:23:33.596022 23319 log.cc:1196] T 59d8302bb4304f698ceb14eb3791d7ce P d7176d64a44646739c4397e808f6cfd1: Deleting WAL directory at /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790626764250411-3144-0/minicluster-data/ts-2/wal/wals/59d8302bb4304f698ceb14eb3791d7ce
I20260928 20:23:33.596174 23318 log.cc:1196] T 59d8302bb4304f698ceb14eb3791d7ce P 2ae7fc60171f49298aba96c606bb2286: Deleting WAL directory at /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790626764250411-3144-0/minicluster-data/ts-1/wal/wals/59d8302bb4304f698ceb14eb3791d7ce
I20260928 20:23:33.596441 23319 ts_tablet_manager.cc:1939] T 59d8302bb4304f698ceb14eb3791d7ce P d7176d64a44646739c4397e808f6cfd1: Deleting consensus metadata
I20260928 20:23:33.596599 23318 ts_tablet_manager.cc:1939] T 59d8302bb4304f698ceb14eb3791d7ce P 2ae7fc60171f49298aba96c606bb2286: Deleting consensus metadata
I20260928 20:23:33.596976 22817 catalog_manager.cc:4841] TS d7176d64a44646739c4397e808f6cfd1 (127.3.18.3:37083): tablet 59d8302bb4304f698ceb14eb3791d7ce (table test-table [id=74cfbc96199b416dbd5b3a011fa3833d]) successfully deleted
I20260928 20:23:33.597067 22816 catalog_manager.cc:4841] TS 2ae7fc60171f49298aba96c606bb2286 (127.3.18.2:33469): tablet 59d8302bb4304f698ceb14eb3791d7ce (table test-table [id=74cfbc96199b416dbd5b3a011fa3833d]) successfully deleted
W20260928 20:23:33.598276 22814 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.18.1:43897 (error 108)
I20260928 20:23:33.598531 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 23021
I20260928 20:23:33.600935 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:33.600671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.1:43897 (local address 127.0.0.1:52604)
0928 20:23:33.600838 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:33.600895 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.1:43897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
W20260928 20:23:33.601084 22814 catalog_manager.cc:4584] TS af19c8a20ffb4b92841fd89231c562ba (127.3.18.1:43897): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 59d8302bb4304f698ceb14eb3791d7ce: Network error: Client connection negotiation failed: client connection to 127.3.18.1:43897: connect: Connection refused (error 111)
I20260928 20:23:33.603643 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 23152
I20260928 20:23:33.608592 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 22799
2026-09-28T20:23:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (872 ms)
[----------] 6 tests from EncryptionPolicyTest (8735 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790626764250411-3144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:23:33 dist-test-slave-b83v krb5kdc[23324](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:23:33 dist-test-slave-b83v krb5kdc[23324](info): set up 2 sockets
Sep 28 20:23:33 dist-test-slave-b83v krb5kdc[23324](info): commencing operation
krb5kdc: starting...
W20260928 20:23:34.673080 3144 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Sep 28 20:23:34 dist-test-slave-b83v krb5kdc[23324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:23:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:23:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790626764250411-3144-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790626764250411-3144-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 20:23:34.802330 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790626764250411-3144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790626764250411-3144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790626764250411-3144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43031
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790626764250411-3144-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43031
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:23:34.887296 23337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:23:34.887571 23337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:23:34.887614 23337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:23:34.888885 23337 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:23:34.888969 23337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:23:34.888993 23337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:23:34.889014 23337 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:23:34.889042 23337 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:23:34.890627 23337 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790626764250411-3144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790626764250411-3144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43031
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790626764250411-3144-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790626764250411-3144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:23:34.890923 23337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:23:34.891167 23337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:23:34.896580 23342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:23:34.896714 23337 server_base.cc:1034] running on GCE node
W20260928 20:23:34.896701 23343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:34.896701 23345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:23:34.897158 23337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:23:34.897500 23337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:23:34.898780 23337 hybrid_clock.cc:648] HybridClock initialized: now 1790627014898733 us; error 58 us; skew 500 ppm
Sep 28 20:23:34 dist-test-slave-b83v krb5kdc[23324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627014, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:23:34.901510 23337 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 20:23:34.902074 23337 webserver.cc:460] Webserver started at http://127.0.0.1:39285/ using document root <none> and password file <none>
I20260928 20:23:34.902314 23337 fs_manager.cc:359] Metadata directory not provided
I20260928 20:23:34.902385 23337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:23:34.902539 23337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:23:34.903637 23337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790626764250411-3144-0/minicluster-data/master-0/data/instance:
uuid: "7e574f3c52e64487a46f7b8d68719e97"
format_stamp: "Formatted at 2026-09-28 20:23:34 on dist-test-slave-b83v"
server_key: "1507fec5f9828fb9afe3c8ef3714b5ad"
server_key_iv: "a992b0382f948a883320d2ff7eb48285"
server_key_version: "encryptionkey@0"
I20260928 20:23:34.904094 23337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790626764250411-3144-0/minicluster-data/master-0/wal/instance:
uuid: "7e574f3c52e64487a46f7b8d68719e97"
format_stamp: "Formatted at 2026-09-28 20:23:34 on dist-test-slave-b83v"
server_key: "1507fec5f9828fb9afe3c8ef3714b5ad"
server_key_iv: "a992b0382f948a883320d2ff7eb48285"
server_key_version: "encryptionkey@0"
I20260928 20:23:34.905586 23337 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:23:34.906513 23352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:34.906816 23337 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:23:34.906962 23337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790626764250411-3144-0/minicluster-data/master-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790626764250411-3144-0/minicluster-data/master-0/wal
uuid: "7e574f3c52e64487a46f7b8d68719e97"
format_stamp: "Formatted at 2026-09-28 20:23:34 on dist-test-slave-b83v"
server_key: "1507fec5f9828fb9afe3c8ef3714b5ad"
server_key_iv: "a992b0382f948a883320d2ff7eb48285"
server_key_version: "encryptionkey@0"
I20260928 20:23:34.907073 23337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790626764250411-3144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790626764250411-3144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790626764250411-3144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:23:34.919492 23337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:23:34.922211 23337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:23:34.922432 23337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:23:34.927446 23337 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43031
I20260928 20:23:34.927521 23404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43031 every 8 connection(s)
I20260928 20:23:34.927958 23337 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
I20260928 20:23:34.929097 23405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:23:34.931499 23405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e574f3c52e64487a46f7b8d68719e97: Bootstrap starting.
I20260928 20:23:34.932142 23405 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e574f3c52e64487a46f7b8d68719e97: Neither blocks nor log segments found. Creating new log.
I20260928 20:23:34.932401 23405 log.cc:824] T 00000000000000000000000000000000 P 7e574f3c52e64487a46f7b8d68719e97: Log is configured to *not* fsync() on all Append() calls
I20260928 20:23:34.933197 23405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e574f3c52e64487a46f7b8d68719e97: No bootstrap required, opened a new log
I20260928 20:23:34.934815 23405 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e574f3c52e64487a46f7b8d68719e97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e574f3c52e64487a46f7b8d68719e97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43031 } }
I20260928 20:23:34.934971 23405 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e574f3c52e64487a46f7b8d68719e97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:23:34.934993 23405 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e574f3c52e64487a46f7b8d68719e97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e574f3c52e64487a46f7b8d68719e97, State: Initialized, Role: FOLLOWER
I20260928 20:23:34.935168 23405 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e574f3c52e64487a46f7b8d68719e97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e574f3c52e64487a46f7b8d68719e97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43031 } }
I20260928 20:23:34.935258 23405 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e574f3c52e64487a46f7b8d68719e97 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:23:34.935283 23405 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e574f3c52e64487a46f7b8d68719e97 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:23:34.935313 23405 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e574f3c52e64487a46f7b8d68719e97 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:23:34.935844 23405 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e574f3c52e64487a46f7b8d68719e97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e574f3c52e64487a46f7b8d68719e97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43031 } }
I20260928 20:23:34.935943 23405 leader_election.cc:302] T 00000000000000000000000000000000 P 7e574f3c52e64487a46f7b8d68719e97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e574f3c52e64487a46f7b8d68719e97; no voters:
I20260928 20:23:34.936092 23405 leader_election.cc:288] T 00000000000000000000000000000000 P 7e574f3c52e64487a46f7b8d68719e97 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:23:34.936160 23408 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e574f3c52e64487a46f7b8d68719e97 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:23:34.936318 23408 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e574f3c52e64487a46f7b8d68719e97 [term 1 LEADER]: Becoming Leader. State: Replica: 7e574f3c52e64487a46f7b8d68719e97, State: Running, Role: LEADER
I20260928 20:23:34.936383 23405 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e574f3c52e64487a46f7b8d68719e97 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:23:34.936407 23408 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e574f3c52e64487a46f7b8d68719e97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e574f3c52e64487a46f7b8d68719e97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43031 } }
I20260928 20:23:34.936851 23409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e574f3c52e64487a46f7b8d68719e97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e574f3c52e64487a46f7b8d68719e97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e574f3c52e64487a46f7b8d68719e97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43031 } } }
I20260928 20:23:34.936942 23409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e574f3c52e64487a46f7b8d68719e97 [sys.catalog]: This master's current role is: LEADER
I20260928 20:23:34.936988 23410 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e574f3c52e64487a46f7b8d68719e97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e574f3c52e64487a46f7b8d68719e97. Latest consensus state: current_term: 1 leader_uuid: "7e574f3c52e64487a46f7b8d68719e97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e574f3c52e64487a46f7b8d68719e97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43031 } } }
I20260928 20:23:34.937178 23410 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e574f3c52e64487a46f7b8d68719e97 [sys.catalog]: This master's current role is: LEADER
I20260928 20:23:34.937376 23416 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:23:34.937908 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 23337
I20260928 20:23:34.938031 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790626764250411-3144-0/minicluster-data/master-0/wal/instance
I20260928 20:23:34.938207 23416 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:23:34.938256 3144 external_mini_cluster.cc:1428] Setting key 3f2dd4efd3a8a59385c9e2c51d3e9f87
I20260928 20:23:34.940289 23416 catalog_manager.cc:1347] Generated new cluster ID: 491183a6151f40a78ca9962a3718bef8
I20260928 20:23:34.940343 23416 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 20:23:34 dist-test-slave-b83v krb5kdc[23324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 20:23:34.950337 23427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:34.938726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53843 (local address 127.0.0.1:43031)
0928 20:23:34.938916 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:34.938926 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:34.938962 (+ 36us) server_negotiation.cc:407] Connection header received
0928 20:23:34.939092 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:34.939098 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:34.939142 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:34.939296 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:34.940259 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:34.941675 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:34.942815 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:34.943435 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:34.943535 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:34.946398 (+ 2863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:34.946403 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:34.946403 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:23:34.946418 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:34.948549 (+ 2131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:34.948997 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:34.949000 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:34.949001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:34.949132 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:34.949384 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:34.949384 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:34.949385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:34.949719 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:34.949883 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:34.949991 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 20:23:34.950093 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":96,"threads_started":1}
I20260928 20:23:34.951305 3144 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 20:23:34.967423 23416 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:23:34.968233 23416 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:23:34.976372 23427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:34.952774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58477 (local address 127.0.0.1:43031)
0928 20:23:34.959388 (+ 6614us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:34.959390 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:34.966329 (+ 6939us) server_negotiation.cc:407] Connection header received
0928 20:23:34.966464 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:34.966465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:34.966479 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:34.966515 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:34.967411 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:34.969040 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:34.970233 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:34.970589 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:34.970682 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:34.972712 (+ 2030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:34.972714 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:34.972715 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:23:34.972725 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:34.974664 (+ 1939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:34.975252 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:34.975254 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:34.975255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:34.975324 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:34.975607 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:34.975608 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:34.975608 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:34.975754 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:34.975901 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:34.976185 (+ 284us) server_negotiation.cc:299] Negotiation successful
0928 20:23:34.976264 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6580}
I20260928 20:23:34.984958 23416 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e574f3c52e64487a46f7b8d68719e97: Generated new TSK 0
I20260928 20:23:34.985159 23416 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 20:23:34.994788 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 23337
W20260928 20:23:34.999439 23322 connection.cc:535] client connection to 127.0.0.1:43031 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43031): Connection reset by peer (error 104)
2026-09-28T20:23:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1382 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790626764250411-3144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:23:35 dist-test-slave-b83v krb5kdc[23436](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:23:35 dist-test-slave-b83v krb5kdc[23436](info): set up 2 sockets
Sep 28 20:23:35 dist-test-slave-b83v krb5kdc[23436](info): commencing operation
krb5kdc: starting...
W20260928 20:23:36.060962 3144 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Sep 28 20:23:36 dist-test-slave-b83v krb5kdc[23436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:23:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:23:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790626764250411-3144-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790626764250411-3144-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 20:23:36.188285 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790626764250411-3144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790626764250411-3144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790626764250411-3144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38769
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790626764250411-3144-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38769
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:23:36.272753 23449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:23:36.273056 23449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:23:36.273099 23449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:23:36.274288 23449 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:23:36.274374 23449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:23:36.274403 23449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:23:36.274437 23449 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:23:36.274493 23449 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:23:36.276057 23449 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790626764250411-3144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790626764250411-3144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38769
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790626764250411-3144-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790626764250411-3144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:23:36.276363 23449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:23:36.276692 23449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:23:36.282162 23457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:36.282158 23455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:23:36.282399 23449 server_base.cc:1034] running on GCE node
W20260928 20:23:36.282253 23454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:23:36.282837 23449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:23:36.283215 23449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:23:36.284431 23449 hybrid_clock.cc:648] HybridClock initialized: now 1790627016284390 us; error 112 us; skew 500 ppm
Sep 28 20:23:36 dist-test-slave-b83v krb5kdc[23436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627016, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:23:36.287608 23449 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 20:23:36.288272 23449 webserver.cc:460] Webserver started at http://127.0.0.1:38141/ using document root <none> and password file <none>
I20260928 20:23:36.288558 23449 fs_manager.cc:359] Metadata directory not provided
I20260928 20:23:36.288640 23449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:23:36.288805 23449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:23:36.290030 23449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790626764250411-3144-0/minicluster-data/master-0/data/instance:
uuid: "4b6519a0d32a4a00ad3faf4bec06cde0"
format_stamp: "Formatted at 2026-09-28 20:23:36 on dist-test-slave-b83v"
server_key: "6fac3b666080cf68281bd5deaff65cc4"
server_key_iv: "66798ad401ca803cc67d057ef3eb5a24"
server_key_version: "encryptionkey@0"
I20260928 20:23:36.290444 23449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790626764250411-3144-0/minicluster-data/master-0/wal/instance:
uuid: "4b6519a0d32a4a00ad3faf4bec06cde0"
format_stamp: "Formatted at 2026-09-28 20:23:36 on dist-test-slave-b83v"
server_key: "6fac3b666080cf68281bd5deaff65cc4"
server_key_iv: "66798ad401ca803cc67d057ef3eb5a24"
server_key_version: "encryptionkey@0"
I20260928 20:23:36.292109 23449 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 20:23:36.293191 23464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:36.293601 23449 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:23:36.293735 23449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790626764250411-3144-0/minicluster-data/master-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790626764250411-3144-0/minicluster-data/master-0/wal
uuid: "4b6519a0d32a4a00ad3faf4bec06cde0"
format_stamp: "Formatted at 2026-09-28 20:23:36 on dist-test-slave-b83v"
server_key: "6fac3b666080cf68281bd5deaff65cc4"
server_key_iv: "66798ad401ca803cc67d057ef3eb5a24"
server_key_version: "encryptionkey@0"
I20260928 20:23:36.293825 23449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790626764250411-3144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790626764250411-3144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790626764250411-3144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:23:36.307376 23449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:23:36.310149 23449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:23:36.310338 23449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:23:36.314713 23449 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38769
I20260928 20:23:36.314754 23516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38769 every 8 connection(s)
I20260928 20:23:36.315125 23449 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
I20260928 20:23:36.316356 23517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:23:36.318907 23517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b6519a0d32a4a00ad3faf4bec06cde0: Bootstrap starting.
I20260928 20:23:36.319563 23517 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b6519a0d32a4a00ad3faf4bec06cde0: Neither blocks nor log segments found. Creating new log.
I20260928 20:23:36.319913 23517 log.cc:824] T 00000000000000000000000000000000 P 4b6519a0d32a4a00ad3faf4bec06cde0: Log is configured to *not* fsync() on all Append() calls
I20260928 20:23:36.320765 23517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b6519a0d32a4a00ad3faf4bec06cde0: No bootstrap required, opened a new log
I20260928 20:23:36.322479 23517 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b6519a0d32a4a00ad3faf4bec06cde0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b6519a0d32a4a00ad3faf4bec06cde0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38769 } }
I20260928 20:23:36.322655 23517 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b6519a0d32a4a00ad3faf4bec06cde0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:23:36.322681 23517 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b6519a0d32a4a00ad3faf4bec06cde0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b6519a0d32a4a00ad3faf4bec06cde0, State: Initialized, Role: FOLLOWER
I20260928 20:23:36.322885 23517 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b6519a0d32a4a00ad3faf4bec06cde0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b6519a0d32a4a00ad3faf4bec06cde0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38769 } }
I20260928 20:23:36.322992 23517 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b6519a0d32a4a00ad3faf4bec06cde0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:23:36.323019 23517 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b6519a0d32a4a00ad3faf4bec06cde0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:23:36.323051 23517 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b6519a0d32a4a00ad3faf4bec06cde0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:23:36.323712 23517 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b6519a0d32a4a00ad3faf4bec06cde0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b6519a0d32a4a00ad3faf4bec06cde0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38769 } }
I20260928 20:23:36.323910 23517 leader_election.cc:302] T 00000000000000000000000000000000 P 4b6519a0d32a4a00ad3faf4bec06cde0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b6519a0d32a4a00ad3faf4bec06cde0; no voters:
I20260928 20:23:36.324128 23517 leader_election.cc:288] T 00000000000000000000000000000000 P 4b6519a0d32a4a00ad3faf4bec06cde0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:23:36.324144 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 23449
I20260928 20:23:36.324208 23520 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b6519a0d32a4a00ad3faf4bec06cde0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:23:36.324271 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790626764250411-3144-0/minicluster-data/master-0/wal/instance
I20260928 20:23:36.324383 23520 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b6519a0d32a4a00ad3faf4bec06cde0 [term 1 LEADER]: Becoming Leader. State: Replica: 4b6519a0d32a4a00ad3faf4bec06cde0, State: Running, Role: LEADER
I20260928 20:23:36.324458 23517 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b6519a0d32a4a00ad3faf4bec06cde0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:23:36.324476 23520 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b6519a0d32a4a00ad3faf4bec06cde0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b6519a0d32a4a00ad3faf4bec06cde0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38769 } }
I20260928 20:23:36.324513 3144 external_mini_cluster.cc:1428] Setting key 4586114c4aaae5420231fff485dc76ee
I20260928 20:23:36.324810 23520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b6519a0d32a4a00ad3faf4bec06cde0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b6519a0d32a4a00ad3faf4bec06cde0. Latest consensus state: current_term: 1 leader_uuid: "4b6519a0d32a4a00ad3faf4bec06cde0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b6519a0d32a4a00ad3faf4bec06cde0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38769 } } }
I20260928 20:23:36.324909 23520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b6519a0d32a4a00ad3faf4bec06cde0 [sys.catalog]: This master's current role is: LEADER
I20260928 20:23:36.324995 23521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b6519a0d32a4a00ad3faf4bec06cde0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b6519a0d32a4a00ad3faf4bec06cde0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b6519a0d32a4a00ad3faf4bec06cde0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38769 } } }
I20260928 20:23:36.325050 23521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b6519a0d32a4a00ad3faf4bec06cde0 [sys.catalog]: This master's current role is: LEADER
I20260928 20:23:36.325322 23525 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:23:36.325929 23525 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:23:36.327916 23525 catalog_manager.cc:1347] Generated new cluster ID: 6593f3006b4943fbbe2bcbe183538204
I20260928 20:23:36.327975 23525 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:23:36.331207 23531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:36.325238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59065 (local address 127.0.0.1:38769)
0928 20:23:36.325558 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:36.325570 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:36.325626 (+ 56us) server_negotiation.cc:407] Connection header received
0928 20:23:36.325792 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:36.325800 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:36.325874 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:36.326389 (+ 515us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:36.326499 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:36.327998 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:36.328981 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:36.329480 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:36.329628 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:36.330438 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:36.330443 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:36.330444 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:23:36.330649 (+ 205us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:36.330684 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:23:36.330709 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:36.330812 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:36.330904 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 20:23:36.330939 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":85,"threads_started":1}
I20260928 20:23:36.332362 3144 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 20:23:36.337242 23531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:36.333940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54155 (local address 127.0.0.1:38769)
0928 20:23:36.333981 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:36.333983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:36.334099 (+ 116us) server_negotiation.cc:407] Connection header received
0928 20:23:36.334186 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:36.334187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:36.334199 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:36.334256 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:36.334417 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:36.334949 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:36.335731 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:36.336022 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:36.336118 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:36.336932 (+ 814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:36.336934 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:36.336934 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:23:36.336940 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:36.336955 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:23:36.336961 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:36.337026 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:36.337110 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 20:23:36.337125 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 20:23:36.347862 23525 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:23:36.348502 23525 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:23:36.357823 23525 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b6519a0d32a4a00ad3faf4bec06cde0: Generated new TSK 0
I20260928 20:23:36.358047 23525 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 20:23:36.362895 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 23449
W20260928 20:23:36.367655 23434 connection.cc:535] client connection to 127.0.0.1:38769 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38769): Connection reset by peer (error 104)
2026-09-28T20:23:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1368 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790626764250411-3144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:23:36 dist-test-slave-b83v krb5kdc[23548](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:23:36 dist-test-slave-b83v krb5kdc[23548](info): set up 2 sockets
Sep 28 20:23:36 dist-test-slave-b83v krb5kdc[23548](info): commencing operation
krb5kdc: starting...
W20260928 20:23:37.424181 3144 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Sep 28 20:23:37 dist-test-slave-b83v krb5kdc[23548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:23:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:23:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790626764250411-3144-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790626764250411-3144-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 20:23:37.554868 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790626764250411-3144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790626764250411-3144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790626764250411-3144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34777
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790626764250411-3144-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:23:37.638113 23561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:23:37.638402 23561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:23:37.638450 23561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:23:37.639703 23561 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:23:37.639806 23561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:23:37.639835 23561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:23:37.639864 23561 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:23:37.639887 23561 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:23:37.641388 23561 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790626764250411-3144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790626764250411-3144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34777
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790626764250411-3144-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23561
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790626764250411-3144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:23:37.641690 23561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:23:37.641932 23561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:23:37.647132 23567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:37.647113 23566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:37.647115 23569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:23:37.647502 23561 server_base.cc:1034] running on GCE node
I20260928 20:23:37.647841 23561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:23:37.648150 23561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:23:37.649358 23561 hybrid_clock.cc:648] HybridClock initialized: now 1790627017649334 us; error 50 us; skew 500 ppm
Sep 28 20:23:37 dist-test-slave-b83v krb5kdc[23548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627017, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:23:37.662047 23561 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 20:23:37.662590 23561 webserver.cc:460] Webserver started at http://127.0.0.1:44433/ using document root <none> and password file <none>
I20260928 20:23:37.662813 23561 fs_manager.cc:359] Metadata directory not provided
I20260928 20:23:37.662858 23561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:23:37.662956 23561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:23:37.664142 23561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790626764250411-3144-0/minicluster-data/master-0/data/instance:
uuid: "1a2f00558394430b9963787cbe6a140e"
format_stamp: "Formatted at 2026-09-28 20:23:37 on dist-test-slave-b83v"
server_key: "c736397fa367940b7511ed61a22c5346"
server_key_iv: "73cf5fff72c9717c98677ebd25b1abc3"
server_key_version: "encryptionkey@0"
I20260928 20:23:37.664477 23561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790626764250411-3144-0/minicluster-data/master-0/wal/instance:
uuid: "1a2f00558394430b9963787cbe6a140e"
format_stamp: "Formatted at 2026-09-28 20:23:37 on dist-test-slave-b83v"
server_key: "c736397fa367940b7511ed61a22c5346"
server_key_iv: "73cf5fff72c9717c98677ebd25b1abc3"
server_key_version: "encryptionkey@0"
I20260928 20:23:37.665849 23561 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 20:23:37.666688 23576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:37.666923 23561 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:23:37.667021 23561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790626764250411-3144-0/minicluster-data/master-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790626764250411-3144-0/minicluster-data/master-0/wal
uuid: "1a2f00558394430b9963787cbe6a140e"
format_stamp: "Formatted at 2026-09-28 20:23:37 on dist-test-slave-b83v"
server_key: "c736397fa367940b7511ed61a22c5346"
server_key_iv: "73cf5fff72c9717c98677ebd25b1abc3"
server_key_version: "encryptionkey@0"
I20260928 20:23:37.667080 23561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790626764250411-3144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790626764250411-3144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790626764250411-3144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:23:37.670665 23561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:23:37.673336 23561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:23:37.673523 23561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:23:37.677706 23561 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34777
I20260928 20:23:37.677757 23628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34777 every 8 connection(s)
I20260928 20:23:37.678143 23561 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
I20260928 20:23:37.679355 23629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:23:37.680459 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 23561
I20260928 20:23:37.680603 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790626764250411-3144-0/minicluster-data/master-0/wal/instance
I20260928 20:23:37.680821 3144 external_mini_cluster.cc:1428] Setting key ed1c1355894dbe215f3bc74b8806796c
I20260928 20:23:37.683271 23629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a2f00558394430b9963787cbe6a140e: Bootstrap starting.
I20260928 20:23:37.684165 23629 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a2f00558394430b9963787cbe6a140e: Neither blocks nor log segments found. Creating new log.
I20260928 20:23:37.684572 23629 log.cc:824] T 00000000000000000000000000000000 P 1a2f00558394430b9963787cbe6a140e: Log is configured to *not* fsync() on all Append() calls
I20260928 20:23:37.685021 23632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:37.681986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59385 (local address 127.0.0.1:34777)
0928 20:23:37.682794 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:37.682805 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:37.682844 (+ 39us) server_negotiation.cc:407] Connection header received
0928 20:23:37.683366 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:37.683374 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:37.683439 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:37.683657 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:37.684519 (+ 862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:37.684523 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:37.684523 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:23:37.684527 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:37.684565 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:23:37.684576 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:37.684652 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:37.684744 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 20:23:37.684798 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":716,"thread_start_us":85,"threads_started":1}
I20260928 20:23:37.685729 23629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a2f00558394430b9963787cbe6a140e: No bootstrap required, opened a new log
I20260928 20:23:37.686182 3144 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 20:23:37.688234 23629 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a2f00558394430b9963787cbe6a140e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a2f00558394430b9963787cbe6a140e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34777 } }
I20260928 20:23:37.688472 23629 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a2f00558394430b9963787cbe6a140e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:23:37.688521 23629 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a2f00558394430b9963787cbe6a140e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a2f00558394430b9963787cbe6a140e, State: Initialized, Role: FOLLOWER
I20260928 20:23:37.688681 23629 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a2f00558394430b9963787cbe6a140e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a2f00558394430b9963787cbe6a140e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34777 } }
I20260928 20:23:37.688812 23629 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a2f00558394430b9963787cbe6a140e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:23:37.688863 23629 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a2f00558394430b9963787cbe6a140e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:23:37.688928 23629 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a2f00558394430b9963787cbe6a140e [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:23:37.689185 23632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:37.687735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60509 (local address 127.0.0.1:34777)
0928 20:23:37.687826 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:37.687829 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:37.687931 (+ 102us) server_negotiation.cc:407] Connection header received
0928 20:23:37.687973 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:37.687974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:37.687979 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:37.688049 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:37.688834 (+ 785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:37.688836 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:37.688836 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:23:37.688842 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:37.688859 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:23:37.688864 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:37.688932 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:37.689048 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 20:23:37.689067 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 20:23:37.689975 23629 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a2f00558394430b9963787cbe6a140e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a2f00558394430b9963787cbe6a140e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34777 } }
I20260928 20:23:37.690124 23629 leader_election.cc:302] T 00000000000000000000000000000000 P 1a2f00558394430b9963787cbe6a140e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a2f00558394430b9963787cbe6a140e; no voters:
I20260928 20:23:37.690342 23629 leader_election.cc:288] T 00000000000000000000000000000000 P 1a2f00558394430b9963787cbe6a140e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:23:37.690397 23639 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a2f00558394430b9963787cbe6a140e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:23:37.690605 23639 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a2f00558394430b9963787cbe6a140e [term 1 LEADER]: Becoming Leader. State: Replica: 1a2f00558394430b9963787cbe6a140e, State: Running, Role: LEADER
I20260928 20:23:37.690699 23629 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a2f00558394430b9963787cbe6a140e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:23:37.690766 23639 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a2f00558394430b9963787cbe6a140e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a2f00558394430b9963787cbe6a140e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34777 } }
I20260928 20:23:37.691196 23640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a2f00558394430b9963787cbe6a140e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a2f00558394430b9963787cbe6a140e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a2f00558394430b9963787cbe6a140e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34777 } } }
I20260928 20:23:37.691196 23641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a2f00558394430b9963787cbe6a140e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a2f00558394430b9963787cbe6a140e. Latest consensus state: current_term: 1 leader_uuid: "1a2f00558394430b9963787cbe6a140e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a2f00558394430b9963787cbe6a140e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34777 } } }
I20260928 20:23:37.691344 23640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a2f00558394430b9963787cbe6a140e [sys.catalog]: This master's current role is: LEADER
I20260928 20:23:37.691414 23641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a2f00558394430b9963787cbe6a140e [sys.catalog]: This master's current role is: LEADER
I20260928 20:23:37.691640 23648 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:23:37.692421 23648 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:23:37.694010 23648 catalog_manager.cc:1347] Generated new cluster ID: e5ea3c2a1267446980486d4fcffbd064
I20260928 20:23:37.694067 23648 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:23:37.718721 23648 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:23:37.719035 23648 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:23:37.724272 23648 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a2f00558394430b9963787cbe6a140e: Generated new TSK 0
I20260928 20:23:37.724524 23648 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 20:23:37.728837 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 23561
2026-09-28T20:23:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1366 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790626764250411-3144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:23:37 dist-test-slave-b83v krb5kdc[23660](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:23:37 dist-test-slave-b83v krb5kdc[23660](info): set up 2 sockets
Sep 28 20:23:37 dist-test-slave-b83v krb5kdc[23660](info): commencing operation
krb5kdc: starting...
W20260928 20:23:38.780129 3144 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Sep 28 20:23:38 dist-test-slave-b83v krb5kdc[23660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:23:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:23:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790626764250411-3144-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790626764250411-3144-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 20:23:38.907750 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790626764250411-3144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790626764250411-3144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790626764250411-3144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34943
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790626764250411-3144-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34943
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:23:38.994926 23673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:23:38.995157 23673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:23:38.995182 23673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:23:38.996323 23673 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:23:38.996358 23673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:23:38.996373 23673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:23:38.996385 23673 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:23:38.996397 23673 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:23:38.997831 23673 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790626764250411-3144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790626764250411-3144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34943
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790626764250411-3144-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790626764250411-3144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:23:38.998035 23673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:23:38.998212 23673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:23:39.003297 23678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:39.003453 23681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:39.003293 23679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:23:39.003559 23673 server_base.cc:1034] running on GCE node
I20260928 20:23:39.003902 23673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:23:39.004158 23673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:23:39.005299 23673 hybrid_clock.cc:648] HybridClock initialized: now 1790627019005286 us; error 30 us; skew 500 ppm
Sep 28 20:23:39 dist-test-slave-b83v krb5kdc[23660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627019, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:23:39.008016 23673 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 20:23:39.008553 23673 webserver.cc:460] Webserver started at http://127.0.0.1:37459/ using document root <none> and password file <none>
I20260928 20:23:39.008821 23673 fs_manager.cc:359] Metadata directory not provided
I20260928 20:23:39.008891 23673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:23:39.009038 23673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:23:39.010123 23673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790626764250411-3144-0/minicluster-data/master-0/data/instance:
uuid: "7895ba41388649e9b56470acfcd5a99d"
format_stamp: "Formatted at 2026-09-28 20:23:39 on dist-test-slave-b83v"
server_key: "2bc1026a25f5f87f961c42dee7d01b82"
server_key_iv: "08c4d73f4402cfc16e870a667a86b743"
server_key_version: "encryptionkey@0"
I20260928 20:23:39.010493 23673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790626764250411-3144-0/minicluster-data/master-0/wal/instance:
uuid: "7895ba41388649e9b56470acfcd5a99d"
format_stamp: "Formatted at 2026-09-28 20:23:39 on dist-test-slave-b83v"
server_key: "2bc1026a25f5f87f961c42dee7d01b82"
server_key_iv: "08c4d73f4402cfc16e870a667a86b743"
server_key_version: "encryptionkey@0"
I20260928 20:23:39.011904 23673 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:23:39.012771 23688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:39.013038 23673 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:23:39.013146 23673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790626764250411-3144-0/minicluster-data/master-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790626764250411-3144-0/minicluster-data/master-0/wal
uuid: "7895ba41388649e9b56470acfcd5a99d"
format_stamp: "Formatted at 2026-09-28 20:23:39 on dist-test-slave-b83v"
server_key: "2bc1026a25f5f87f961c42dee7d01b82"
server_key_iv: "08c4d73f4402cfc16e870a667a86b743"
server_key_version: "encryptionkey@0"
I20260928 20:23:39.013229 23673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790626764250411-3144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790626764250411-3144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790626764250411-3144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:23:39.028967 23673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:23:39.030967 23673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:23:39.031109 23673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:23:39.035311 23673 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34943
I20260928 20:23:39.035369 23740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34943 every 8 connection(s)
I20260928 20:23:39.035704 23673 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
I20260928 20:23:39.036723 23741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:23:39.038997 23741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7895ba41388649e9b56470acfcd5a99d: Bootstrap starting.
I20260928 20:23:39.039518 23741 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7895ba41388649e9b56470acfcd5a99d: Neither blocks nor log segments found. Creating new log.
I20260928 20:23:39.040063 23741 log.cc:824] T 00000000000000000000000000000000 P 7895ba41388649e9b56470acfcd5a99d: Log is configured to *not* fsync() on all Append() calls
I20260928 20:23:39.040827 23741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7895ba41388649e9b56470acfcd5a99d: No bootstrap required, opened a new log
I20260928 20:23:39.042418 23741 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7895ba41388649e9b56470acfcd5a99d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7895ba41388649e9b56470acfcd5a99d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34943 } }
I20260928 20:23:39.042557 23741 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7895ba41388649e9b56470acfcd5a99d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:23:39.042578 23741 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7895ba41388649e9b56470acfcd5a99d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7895ba41388649e9b56470acfcd5a99d, State: Initialized, Role: FOLLOWER
I20260928 20:23:39.042721 23741 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7895ba41388649e9b56470acfcd5a99d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7895ba41388649e9b56470acfcd5a99d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34943 } }
I20260928 20:23:39.042804 23741 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7895ba41388649e9b56470acfcd5a99d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:23:39.042827 23741 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7895ba41388649e9b56470acfcd5a99d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:23:39.042855 23741 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7895ba41388649e9b56470acfcd5a99d [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:23:39.043099 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 23673
I20260928 20:23:39.043228 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790626764250411-3144-0/minicluster-data/master-0/wal/instance
I20260928 20:23:39.043416 3144 external_mini_cluster.cc:1428] Setting key 01eb28400fdfd255bc3668f4cdfa31a8
I20260928 20:23:39.043602 23741 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7895ba41388649e9b56470acfcd5a99d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7895ba41388649e9b56470acfcd5a99d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34943 } }
I20260928 20:23:39.043761 23741 leader_election.cc:302] T 00000000000000000000000000000000 P 7895ba41388649e9b56470acfcd5a99d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7895ba41388649e9b56470acfcd5a99d; no voters:
I20260928 20:23:39.043954 23741 leader_election.cc:288] T 00000000000000000000000000000000 P 7895ba41388649e9b56470acfcd5a99d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:23:39.043982 23744 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7895ba41388649e9b56470acfcd5a99d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:23:39.044202 23744 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7895ba41388649e9b56470acfcd5a99d [term 1 LEADER]: Becoming Leader. State: Replica: 7895ba41388649e9b56470acfcd5a99d, State: Running, Role: LEADER
I20260928 20:23:39.044312 23744 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7895ba41388649e9b56470acfcd5a99d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7895ba41388649e9b56470acfcd5a99d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34943 } }
I20260928 20:23:39.044315 23741 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7895ba41388649e9b56470acfcd5a99d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:23:39.044746 23747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7895ba41388649e9b56470acfcd5a99d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7895ba41388649e9b56470acfcd5a99d. Latest consensus state: current_term: 1 leader_uuid: "7895ba41388649e9b56470acfcd5a99d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7895ba41388649e9b56470acfcd5a99d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34943 } } }
I20260928 20:23:39.044833 23747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7895ba41388649e9b56470acfcd5a99d [sys.catalog]: This master's current role is: LEADER
I20260928 20:23:39.044759 23748 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7895ba41388649e9b56470acfcd5a99d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7895ba41388649e9b56470acfcd5a99d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7895ba41388649e9b56470acfcd5a99d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34943 } } }
I20260928 20:23:39.045092 23748 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7895ba41388649e9b56470acfcd5a99d [sys.catalog]: This master's current role is: LEADER
I20260928 20:23:39.045428 23750 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:23:39.046030 23750 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:23:39.048085 23750 catalog_manager.cc:1347] Generated new cluster ID: da536f26bb4a4ca38c67765dd1cb18c6
I20260928 20:23:39.048141 23750 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 20:23:39 dist-test-slave-b83v krb5kdc[23660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 20:23:39.055265 23746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:39.044031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35443 (local address 127.0.0.1:34943)
0928 20:23:39.044201 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:39.044213 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:39.044758 (+ 545us) server_negotiation.cc:407] Connection header received
0928 20:23:39.044882 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:39.044890 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:39.044954 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:39.045091 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:39.046215 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:39.047595 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:39.048593 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:39.049018 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:39.049148 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:39.051481 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:39.051485 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:39.051485 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:23:39.051497 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:39.053679 (+ 2182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:39.054067 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:39.054070 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:39.054070 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:39.054182 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:39.054407 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:39.054408 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:39.054408 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:39.054647 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:39.054828 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:39.054893 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 20:23:39.055039 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":84,"threads_started":1}
I20260928 20:23:39.056249 3144 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 20:23:39.061668 23750 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:23:39.062439 23750 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:23:39.066543 23746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:39.057621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51285 (local address 127.0.0.1:34943)
0928 20:23:39.057669 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:39.057671 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:39.059962 (+ 2291us) server_negotiation.cc:407] Connection header received
0928 20:23:39.060056 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:39.060057 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:39.060068 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:39.060126 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:39.060734 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:39.061314 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:39.062212 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:39.062431 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:39.062509 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:39.063810 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:39.063812 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:39.063812 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:23:39.063821 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:39.065357 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:39.065733 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:39.065735 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:39.065735 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:39.065798 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:39.066010 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:39.066011 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:39.066011 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:39.066127 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:39.066231 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:39.066341 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 20:23:39.066435 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 20:23:39.070884 23750 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7895ba41388649e9b56470acfcd5a99d: Generated new TSK 0
I20260928 20:23:39.071120 23750 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 20:23:39.078465 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 23673
2026-09-28T20:23:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1348 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790626764250411-3144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:23:39 dist-test-slave-b83v krb5kdc[23772](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:23:39 dist-test-slave-b83v krb5kdc[23772](info): set up 2 sockets
Sep 28 20:23:39 dist-test-slave-b83v krb5kdc[23772](info): commencing operation
krb5kdc: starting...
W20260928 20:23:40.128194 3144 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Sep 28 20:23:40 dist-test-slave-b83v krb5kdc[23772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:23:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:23:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790626764250411-3144-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790626764250411-3144-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 20:23:40.263118 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790626764250411-3144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790626764250411-3144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790626764250411-3144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42537
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790626764250411-3144-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42537
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:23:40.347206 23785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:23:40.347445 23785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:23:40.347469 23785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:23:40.348848 23785 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:23:40.348902 23785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:23:40.348917 23785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:23:40.348930 23785 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:23:40.348942 23785 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:23:40.350569 23785 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790626764250411-3144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790626764250411-3144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42537
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790626764250411-3144-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790626764250411-3144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:23:40.350862 23785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:23:40.351162 23785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:23:40.356531 23790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:40.356539 23791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:23:40.356580 23785 server_base.cc:1034] running on GCE node
W20260928 20:23:40.356657 23793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:23:40.357236 23785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:23:40.357657 23785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:23:40.358866 23785 hybrid_clock.cc:648] HybridClock initialized: now 1790627020358808 us; error 87 us; skew 500 ppm
Sep 28 20:23:40 dist-test-slave-b83v krb5kdc[23772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627020, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:23:40.362003 23785 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 20:23:40.362502 23785 webserver.cc:460] Webserver started at http://127.0.0.1:40209/ using document root <none> and password file <none>
I20260928 20:23:40.362769 23785 fs_manager.cc:359] Metadata directory not provided
I20260928 20:23:40.362843 23785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:23:40.362998 23785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:23:40.364280 23785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790626764250411-3144-0/minicluster-data/master-0/data/instance:
uuid: "5bc4bdeecacd40e597e027f6c95600d6"
format_stamp: "Formatted at 2026-09-28 20:23:40 on dist-test-slave-b83v"
server_key: "226817d3685d3c72ccb8bcd2734ddf64"
server_key_iv: "8b6024747c84462f5f451b52c0f136a5"
server_key_version: "encryptionkey@0"
I20260928 20:23:40.364653 23785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790626764250411-3144-0/minicluster-data/master-0/wal/instance:
uuid: "5bc4bdeecacd40e597e027f6c95600d6"
format_stamp: "Formatted at 2026-09-28 20:23:40 on dist-test-slave-b83v"
server_key: "226817d3685d3c72ccb8bcd2734ddf64"
server_key_iv: "8b6024747c84462f5f451b52c0f136a5"
server_key_version: "encryptionkey@0"
I20260928 20:23:40.366199 23785 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 20:23:40.367130 23800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:40.367396 23785 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:23:40.367512 23785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790626764250411-3144-0/minicluster-data/master-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790626764250411-3144-0/minicluster-data/master-0/wal
uuid: "5bc4bdeecacd40e597e027f6c95600d6"
format_stamp: "Formatted at 2026-09-28 20:23:40 on dist-test-slave-b83v"
server_key: "226817d3685d3c72ccb8bcd2734ddf64"
server_key_iv: "8b6024747c84462f5f451b52c0f136a5"
server_key_version: "encryptionkey@0"
I20260928 20:23:40.367600 23785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790626764250411-3144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790626764250411-3144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790626764250411-3144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:23:40.381582 23785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:23:40.383991 23785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:23:40.384207 23785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:23:40.388185 23785 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42537
I20260928 20:23:40.388238 23852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42537 every 8 connection(s)
I20260928 20:23:40.388693 23785 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
I20260928 20:23:40.389827 23853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:23:40.392391 23853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bc4bdeecacd40e597e027f6c95600d6: Bootstrap starting.
I20260928 20:23:40.393007 23853 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5bc4bdeecacd40e597e027f6c95600d6: Neither blocks nor log segments found. Creating new log.
I20260928 20:23:40.393318 23853 log.cc:824] T 00000000000000000000000000000000 P 5bc4bdeecacd40e597e027f6c95600d6: Log is configured to *not* fsync() on all Append() calls
I20260928 20:23:40.394093 23853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bc4bdeecacd40e597e027f6c95600d6: No bootstrap required, opened a new log
I20260928 20:23:40.395817 23853 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5bc4bdeecacd40e597e027f6c95600d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bc4bdeecacd40e597e027f6c95600d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42537 } }
I20260928 20:23:40.395962 23853 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5bc4bdeecacd40e597e027f6c95600d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:23:40.395984 23853 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5bc4bdeecacd40e597e027f6c95600d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bc4bdeecacd40e597e027f6c95600d6, State: Initialized, Role: FOLLOWER
I20260928 20:23:40.396132 23853 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5bc4bdeecacd40e597e027f6c95600d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bc4bdeecacd40e597e027f6c95600d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42537 } }
I20260928 20:23:40.396226 23853 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5bc4bdeecacd40e597e027f6c95600d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:23:40.396268 23853 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5bc4bdeecacd40e597e027f6c95600d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:23:40.396327 23853 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5bc4bdeecacd40e597e027f6c95600d6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:23:40.396878 23853 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5bc4bdeecacd40e597e027f6c95600d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bc4bdeecacd40e597e027f6c95600d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42537 } }
I20260928 20:23:40.396963 23853 leader_election.cc:302] T 00000000000000000000000000000000 P 5bc4bdeecacd40e597e027f6c95600d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5bc4bdeecacd40e597e027f6c95600d6; no voters:
I20260928 20:23:40.397150 23853 leader_election.cc:288] T 00000000000000000000000000000000 P 5bc4bdeecacd40e597e027f6c95600d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:23:40.397197 23856 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5bc4bdeecacd40e597e027f6c95600d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:23:40.397388 23856 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5bc4bdeecacd40e597e027f6c95600d6 [term 1 LEADER]: Becoming Leader. State: Replica: 5bc4bdeecacd40e597e027f6c95600d6, State: Running, Role: LEADER
I20260928 20:23:40.397434 23853 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5bc4bdeecacd40e597e027f6c95600d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:23:40.397574 23856 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5bc4bdeecacd40e597e027f6c95600d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bc4bdeecacd40e597e027f6c95600d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42537 } }
I20260928 20:23:40.398195 23857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bc4bdeecacd40e597e027f6c95600d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5bc4bdeecacd40e597e027f6c95600d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bc4bdeecacd40e597e027f6c95600d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42537 } } }
I20260928 20:23:40.398212 23858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bc4bdeecacd40e597e027f6c95600d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bc4bdeecacd40e597e027f6c95600d6. Latest consensus state: current_term: 1 leader_uuid: "5bc4bdeecacd40e597e027f6c95600d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bc4bdeecacd40e597e027f6c95600d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42537 } } }
I20260928 20:23:40.398316 23858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bc4bdeecacd40e597e027f6c95600d6 [sys.catalog]: This master's current role is: LEADER
I20260928 20:23:40.398303 23857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bc4bdeecacd40e597e027f6c95600d6 [sys.catalog]: This master's current role is: LEADER
I20260928 20:23:40.398533 23865 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:23:40.398854 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 23785
I20260928 20:23:40.398988 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790626764250411-3144-0/minicluster-data/master-0/wal/instance
I20260928 20:23:40.399147 3144 external_mini_cluster.cc:1428] Setting key 08423df942771658e69296f85967f54e
I20260928 20:23:40.399178 23865 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:23:40.400923 23865 catalog_manager.cc:1347] Generated new cluster ID: 211f17fad87d43f79ec03a53328d1aae
I20260928 20:23:40.400975 23865 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:23:40.405597 23875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:40.399793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47527 (local address 127.0.0.1:42537)
0928 20:23:40.399972 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:40.399982 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:40.400089 (+ 107us) server_negotiation.cc:407] Connection header received
0928 20:23:40.400252 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:40.400258 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:40.400330 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:40.400489 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:40.400875 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:40.402270 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:40.403389 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:40.404039 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:40.404145 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:40.404874 (+ 729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:40.404878 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:40.404878 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:23:40.405080 (+ 202us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:40.405118 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:23:40.405140 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:40.405200 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:40.405328 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 20:23:40.405390 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":98,"threads_started":1}
I20260928 20:23:40.406783 3144 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 20:23:40.411943 23875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:40.408343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59663 (local address 127.0.0.1:42537)
0928 20:23:40.408375 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:40.408377 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:40.408639 (+ 262us) server_negotiation.cc:407] Connection header received
0928 20:23:40.408695 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:40.408696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:40.408706 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:40.408758 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:40.408913 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:40.409633 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:40.410646 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:40.410934 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:40.410991 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:40.411675 (+ 684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:40.411676 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:40.411676 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:23:40.411682 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:40.411696 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:23:40.411700 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:40.411748 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:40.411845 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 20:23:40.411870 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 20:23:40.417148 23865 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:23:40.417900 23865 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:23:40.424515 23865 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5bc4bdeecacd40e597e027f6c95600d6: Generated new TSK 0
I20260928 20:23:40.424719 23865 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 20:23:40.431632 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 23785
2026-09-28T20:23:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1353 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790626764250411-3144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:23:40 dist-test-slave-b83v krb5kdc[23884](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:23:40 dist-test-slave-b83v krb5kdc[23884](info): set up 2 sockets
Sep 28 20:23:40 dist-test-slave-b83v krb5kdc[23884](info): commencing operation
krb5kdc: starting...
W20260928 20:23:41.540237 3144 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Sep 28 20:23:41 dist-test-slave-b83v krb5kdc[23884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:23:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:23:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790626764250411-3144-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790626764250411-3144-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 20:23:41.667711 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790626764250411-3144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790626764250411-3144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790626764250411-3144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43285
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790626764250411-3144-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43285
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:23:41.753105 23897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:23:41.753432 23897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:23:41.753484 23897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:23:41.754855 23897 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:23:41.754956 23897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:23:41.754983 23897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:23:41.755009 23897 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:23:41.755050 23897 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:23:41.756788 23897 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790626764250411-3144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790626764250411-3144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43285
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790626764250411-3144-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790626764250411-3144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:23:41.757141 23897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:23:41.757434 23897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:23:41.763079 23902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:41.763082 23903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:23:41.763305 23897 server_base.cc:1034] running on GCE node
W20260928 20:23:41.763160 23905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:23:41.763705 23897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:23:41.764074 23897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:23:41.765280 23897 hybrid_clock.cc:648] HybridClock initialized: now 1790627021765262 us; error 42 us; skew 500 ppm
Sep 28 20:23:41 dist-test-slave-b83v krb5kdc[23884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627021, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:23:41.768018 23897 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 20:23:41.768491 23897 webserver.cc:460] Webserver started at http://127.0.0.1:37827/ using document root <none> and password file <none>
I20260928 20:23:41.768743 23897 fs_manager.cc:359] Metadata directory not provided
I20260928 20:23:41.768813 23897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:23:41.768980 23897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:23:41.770146 23897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790626764250411-3144-0/minicluster-data/master-0/data/instance:
uuid: "c6aadcfbe7604b3c9d9ead1509decf58"
format_stamp: "Formatted at 2026-09-28 20:23:41 on dist-test-slave-b83v"
server_key: "7d83d11d0bedae8d451e8cd6368976c2"
server_key_iv: "aa335c8f4137c91a1001229719885dd6"
server_key_version: "encryptionkey@0"
I20260928 20:23:41.770547 23897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790626764250411-3144-0/minicluster-data/master-0/wal/instance:
uuid: "c6aadcfbe7604b3c9d9ead1509decf58"
format_stamp: "Formatted at 2026-09-28 20:23:41 on dist-test-slave-b83v"
server_key: "7d83d11d0bedae8d451e8cd6368976c2"
server_key_iv: "aa335c8f4137c91a1001229719885dd6"
server_key_version: "encryptionkey@0"
I20260928 20:23:41.772300 23897 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260928 20:23:41.773249 23912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:41.773458 23897 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:23:41.773567 23897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790626764250411-3144-0/minicluster-data/master-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790626764250411-3144-0/minicluster-data/master-0/wal
uuid: "c6aadcfbe7604b3c9d9ead1509decf58"
format_stamp: "Formatted at 2026-09-28 20:23:41 on dist-test-slave-b83v"
server_key: "7d83d11d0bedae8d451e8cd6368976c2"
server_key_iv: "aa335c8f4137c91a1001229719885dd6"
server_key_version: "encryptionkey@0"
I20260928 20:23:41.773660 23897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790626764250411-3144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790626764250411-3144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790626764250411-3144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:23:41.777264 23897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:23:41.779294 23897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:23:41.779446 23897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:23:41.783571 23897 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43285
I20260928 20:23:41.783633 23964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43285 every 8 connection(s)
I20260928 20:23:41.783993 23897 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
I20260928 20:23:41.785025 23965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:23:41.787365 23965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6aadcfbe7604b3c9d9ead1509decf58: Bootstrap starting.
I20260928 20:23:41.788050 23965 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6aadcfbe7604b3c9d9ead1509decf58: Neither blocks nor log segments found. Creating new log.
I20260928 20:23:41.788329 23965 log.cc:824] T 00000000000000000000000000000000 P c6aadcfbe7604b3c9d9ead1509decf58: Log is configured to *not* fsync() on all Append() calls
I20260928 20:23:41.789068 23965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6aadcfbe7604b3c9d9ead1509decf58: No bootstrap required, opened a new log
I20260928 20:23:41.790736 23965 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6aadcfbe7604b3c9d9ead1509decf58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6aadcfbe7604b3c9d9ead1509decf58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43285 } }
I20260928 20:23:41.790882 23965 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6aadcfbe7604b3c9d9ead1509decf58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:23:41.790905 23965 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6aadcfbe7604b3c9d9ead1509decf58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6aadcfbe7604b3c9d9ead1509decf58, State: Initialized, Role: FOLLOWER
I20260928 20:23:41.791011 23965 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6aadcfbe7604b3c9d9ead1509decf58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6aadcfbe7604b3c9d9ead1509decf58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43285 } }
I20260928 20:23:41.791062 23965 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6aadcfbe7604b3c9d9ead1509decf58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:23:41.791083 23965 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6aadcfbe7604b3c9d9ead1509decf58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:23:41.791162 23965 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6aadcfbe7604b3c9d9ead1509decf58 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:23:41.791723 23965 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6aadcfbe7604b3c9d9ead1509decf58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6aadcfbe7604b3c9d9ead1509decf58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43285 } }
I20260928 20:23:41.791896 23965 leader_election.cc:302] T 00000000000000000000000000000000 P c6aadcfbe7604b3c9d9ead1509decf58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6aadcfbe7604b3c9d9ead1509decf58; no voters:
I20260928 20:23:41.792091 23965 leader_election.cc:288] T 00000000000000000000000000000000 P c6aadcfbe7604b3c9d9ead1509decf58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:23:41.792171 23968 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6aadcfbe7604b3c9d9ead1509decf58 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:23:41.792330 23968 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6aadcfbe7604b3c9d9ead1509decf58 [term 1 LEADER]: Becoming Leader. State: Replica: c6aadcfbe7604b3c9d9ead1509decf58, State: Running, Role: LEADER
I20260928 20:23:41.792362 23965 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6aadcfbe7604b3c9d9ead1509decf58 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:23:41.792500 23968 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6aadcfbe7604b3c9d9ead1509decf58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6aadcfbe7604b3c9d9ead1509decf58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43285 } }
I20260928 20:23:41.792752 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 23897
I20260928 20:23:41.792894 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790626764250411-3144-0/minicluster-data/master-0/wal/instance
I20260928 20:23:41.793110 3144 external_mini_cluster.cc:1428] Setting key 57a9fb3721c784a76f34a6fc1ca35ce8
I20260928 20:23:41.793206 23970 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6aadcfbe7604b3c9d9ead1509decf58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6aadcfbe7604b3c9d9ead1509decf58. Latest consensus state: current_term: 1 leader_uuid: "c6aadcfbe7604b3c9d9ead1509decf58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6aadcfbe7604b3c9d9ead1509decf58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43285 } } }
I20260928 20:23:41.793303 23970 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6aadcfbe7604b3c9d9ead1509decf58 [sys.catalog]: This master's current role is: LEADER
I20260928 20:23:41.793700 23969 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6aadcfbe7604b3c9d9ead1509decf58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6aadcfbe7604b3c9d9ead1509decf58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6aadcfbe7604b3c9d9ead1509decf58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43285 } } }
I20260928 20:23:41.793785 23969 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6aadcfbe7604b3c9d9ead1509decf58 [sys.catalog]: This master's current role is: LEADER
I20260928 20:23:41.793962 23978 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:23:41.794477 23978 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:23:41.796777 23979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:41.793745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38719 (local address 127.0.0.1:43285)
0928 20:23:41.795135 (+ 1390us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:41.795145 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:41.795190 (+ 45us) server_negotiation.cc:407] Connection header received
0928 20:23:41.795304 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:41.795311 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:41.795342 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:41.795494 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:41.796256 (+ 762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:41.796260 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:41.796260 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:23:41.796265 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:41.796310 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:23:41.796326 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:41.796424 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:41.796538 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 20:23:41.796585 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1340,"thread_start_us":182,"threads_started":1}
I20260928 20:23:41.796921 23978 catalog_manager.cc:1347] Generated new cluster ID: 2729ba6dba5445d7a6416cea54ca8200
I20260928 20:23:41.796965 23978 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:23:41.800333 3144 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 20:23:41.803159 23979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:41.802074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48585 (local address 127.0.0.1:43285)
0928 20:23:41.802121 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:41.802124 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:41.802133 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:23:41.802192 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:41.802193 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:41.802199 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:41.802244 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:41.802916 (+ 672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:41.802918 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:41.802919 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:23:41.802923 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:41.802941 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:23:41.802945 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:41.803004 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:41.803062 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 20:23:41.803078 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 20:23:41.807837 23978 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:23:41.808027 23978 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:23:41.812453 23978 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6aadcfbe7604b3c9d9ead1509decf58: Generated new TSK 0
I20260928 20:23:41.812650 23978 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 20:23:41.818359 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 23897
2026-09-28T20:23:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1386 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790626764250411-3144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:23:41 dist-test-slave-b83v krb5kdc[23996](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:23:41 dist-test-slave-b83v krb5kdc[23996](info): set up 2 sockets
Sep 28 20:23:41 dist-test-slave-b83v krb5kdc[23996](info): commencing operation
krb5kdc: starting...
W20260928 20:23:42.868299 3144 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Sep 28 20:23:42 dist-test-slave-b83v krb5kdc[23996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:23:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:23:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790626764250411-3144-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790626764250411-3144-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 20:23:42.996460 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790626764250411-3144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790626764250411-3144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790626764250411-3144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41373
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790626764250411-3144-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41373
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:23:43.080607 24009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:23:43.080849 24009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:23:43.080874 24009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:23:43.082140 24009 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:23:43.082193 24009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:23:43.082207 24009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:23:43.082221 24009 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:23:43.082232 24009 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:23:43.083755 24009 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790626764250411-3144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790626764250411-3144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41373
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790626764250411-3144-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790626764250411-3144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:23:43.084041 24009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:23:43.084283 24009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:23:43.089525 24014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:43.089546 24015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:23:43.089545 24009 server_base.cc:1034] running on GCE node
W20260928 20:23:43.089525 24017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:23:43.090067 24009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:23:43.090402 24009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:23:43.091584 24009 hybrid_clock.cc:648] HybridClock initialized: now 1790627023091571 us; error 45 us; skew 500 ppm
Sep 28 20:23:43 dist-test-slave-b83v krb5kdc[23996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:23:43.101953 24009 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 20:23:43.102448 24009 webserver.cc:460] Webserver started at http://127.0.0.1:33409/ using document root <none> and password file <none>
I20260928 20:23:43.102661 24009 fs_manager.cc:359] Metadata directory not provided
I20260928 20:23:43.102705 24009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:23:43.102797 24009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:23:43.103880 24009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790626764250411-3144-0/minicluster-data/master-0/data/instance:
uuid: "52c6feb363294aa294182d99a5aff49a"
format_stamp: "Formatted at 2026-09-28 20:23:43 on dist-test-slave-b83v"
server_key: "eae91f088b49af3506fe67038d82af54"
server_key_iv: "4cf7114653073f74e31cb0b2385380e2"
server_key_version: "encryptionkey@0"
I20260928 20:23:43.104197 24009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790626764250411-3144-0/minicluster-data/master-0/wal/instance:
uuid: "52c6feb363294aa294182d99a5aff49a"
format_stamp: "Formatted at 2026-09-28 20:23:43 on dist-test-slave-b83v"
server_key: "eae91f088b49af3506fe67038d82af54"
server_key_iv: "4cf7114653073f74e31cb0b2385380e2"
server_key_version: "encryptionkey@0"
I20260928 20:23:43.105432 24009 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:23:43.106129 24024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:43.106345 24009 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:23:43.106434 24009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790626764250411-3144-0/minicluster-data/master-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790626764250411-3144-0/minicluster-data/master-0/wal
uuid: "52c6feb363294aa294182d99a5aff49a"
format_stamp: "Formatted at 2026-09-28 20:23:43 on dist-test-slave-b83v"
server_key: "eae91f088b49af3506fe67038d82af54"
server_key_iv: "4cf7114653073f74e31cb0b2385380e2"
server_key_version: "encryptionkey@0"
I20260928 20:23:43.106487 24009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790626764250411-3144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790626764250411-3144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790626764250411-3144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:23:43.115124 24009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:23:43.117350 24009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:23:43.117470 24009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:23:43.121639 24009 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41373
I20260928 20:23:43.121719 24076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41373 every 8 connection(s)
I20260928 20:23:43.122080 24009 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
I20260928 20:23:43.123121 24077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:23:43.125543 24077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52c6feb363294aa294182d99a5aff49a: Bootstrap starting.
I20260928 20:23:43.126148 24077 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52c6feb363294aa294182d99a5aff49a: Neither blocks nor log segments found. Creating new log.
I20260928 20:23:43.126451 24077 log.cc:824] T 00000000000000000000000000000000 P 52c6feb363294aa294182d99a5aff49a: Log is configured to *not* fsync() on all Append() calls
I20260928 20:23:43.127192 24077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52c6feb363294aa294182d99a5aff49a: No bootstrap required, opened a new log
I20260928 20:23:43.128961 24077 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52c6feb363294aa294182d99a5aff49a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52c6feb363294aa294182d99a5aff49a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41373 } }
I20260928 20:23:43.129130 24077 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52c6feb363294aa294182d99a5aff49a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:23:43.129176 24077 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52c6feb363294aa294182d99a5aff49a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52c6feb363294aa294182d99a5aff49a, State: Initialized, Role: FOLLOWER
I20260928 20:23:43.129362 24077 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52c6feb363294aa294182d99a5aff49a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52c6feb363294aa294182d99a5aff49a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41373 } }
I20260928 20:23:43.129463 24077 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52c6feb363294aa294182d99a5aff49a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:23:43.129511 24077 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52c6feb363294aa294182d99a5aff49a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:23:43.129563 24077 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52c6feb363294aa294182d99a5aff49a [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:23:43.130116 24077 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52c6feb363294aa294182d99a5aff49a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52c6feb363294aa294182d99a5aff49a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41373 } }
I20260928 20:23:43.130239 24077 leader_election.cc:302] T 00000000000000000000000000000000 P 52c6feb363294aa294182d99a5aff49a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52c6feb363294aa294182d99a5aff49a; no voters:
I20260928 20:23:43.130437 24077 leader_election.cc:288] T 00000000000000000000000000000000 P 52c6feb363294aa294182d99a5aff49a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:23:43.130544 24080 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52c6feb363294aa294182d99a5aff49a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:23:43.130709 24077 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52c6feb363294aa294182d99a5aff49a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:23:43.130788 24080 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52c6feb363294aa294182d99a5aff49a [term 1 LEADER]: Becoming Leader. State: Replica: 52c6feb363294aa294182d99a5aff49a, State: Running, Role: LEADER
I20260928 20:23:43.130898 24080 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52c6feb363294aa294182d99a5aff49a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52c6feb363294aa294182d99a5aff49a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41373 } }
I20260928 20:23:43.131575 24082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52c6feb363294aa294182d99a5aff49a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52c6feb363294aa294182d99a5aff49a. Latest consensus state: current_term: 1 leader_uuid: "52c6feb363294aa294182d99a5aff49a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52c6feb363294aa294182d99a5aff49a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41373 } } }
I20260928 20:23:43.131685 24082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52c6feb363294aa294182d99a5aff49a [sys.catalog]: This master's current role is: LEADER
I20260928 20:23:43.131740 24081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52c6feb363294aa294182d99a5aff49a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52c6feb363294aa294182d99a5aff49a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52c6feb363294aa294182d99a5aff49a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41373 } } }
I20260928 20:23:43.131850 24081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52c6feb363294aa294182d99a5aff49a [sys.catalog]: This master's current role is: LEADER
I20260928 20:23:43.132071 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 24009
I20260928 20:23:43.132228 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790626764250411-3144-0/minicluster-data/master-0/wal/instance
I20260928 20:23:43.132432 3144 external_mini_cluster.cc:1428] Setting key c0c33522a163851f2cd44d29a7a8857e
W20260928 20:23:43.132694 24095 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 52c6feb363294aa294182d99a5aff49a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 20:23:43.132768 24095 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 20:23:43.132866 24091 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:23:43.133065 24091 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:23:43.134753 24091 catalog_manager.cc:1347] Generated new cluster ID: b1f595533db64ff4b9b6c91d5fb36539
I20260928 20:23:43.134809 24091 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 20:23:43 dist-test-slave-b83v krb5kdc[23996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 20:23:43.143206 24099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:43.132934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.165:40273 (local address 127.0.0.1:41373)
0928 20:23:43.133138 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:43.133148 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:43.133195 (+ 47us) server_negotiation.cc:407] Connection header received
0928 20:23:43.133316 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:43.133324 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:43.133383 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:43.133549 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:43.134354 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:43.135431 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:43.136361 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:43.136843 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:43.136977 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:43.139549 (+ 2572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:43.139553 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:43.139554 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:23:43.139576 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:43.141453 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:43.141893 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:43.141903 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:43.141903 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:43.142019 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:43.142265 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:43.142266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:43.142266 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:43.142536 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:43.142729 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:43.142861 (+ 132us) server_negotiation.cc:299] Negotiation successful
0928 20:23:43.142975 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":79,"threads_started":1}
I20260928 20:23:43.144405 3144 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 20:23:43.155349 24099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:43.145928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.165:45621 (local address 127.0.0.1:41373)
0928 20:23:43.145990 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:43.145992 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:43.148005 (+ 2013us) server_negotiation.cc:407] Connection header received
0928 20:23:43.148094 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:43.148095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:43.148106 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:43.148163 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:43.148831 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:43.149410 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:43.150157 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:43.150392 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:43.150489 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:43.151873 (+ 1384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:43.151874 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:43.151875 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:23:43.151883 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:43.153997 (+ 2114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:43.154427 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:43.154428 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:43.154429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:43.154493 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:43.154746 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:43.154748 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:43.154748 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:43.154872 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:43.154991 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:43.155140 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 20:23:43.155208 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 20:23:43.165284 24091 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:23:43.166078 24091 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:23:43.189033 24091 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52c6feb363294aa294182d99a5aff49a: Generated new TSK 0
I20260928 20:23:43.189309 24091 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 20:23:43.195713 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 24009
W20260928 20:23:43.200819 23994 connection.cc:535] client connection to 127.0.0.1:41373 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41373): Connection reset by peer (error 104)
2026-09-28T20:23:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1378 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790626764250411-3144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:23:43 dist-test-slave-b83v krb5kdc[24108](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:23:43 dist-test-slave-b83v krb5kdc[24108](info): set up 2 sockets
Sep 28 20:23:43 dist-test-slave-b83v krb5kdc[24108](info): commencing operation
krb5kdc: starting...
W20260928 20:23:44.260358 3144 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Sep 28 20:23:44 dist-test-slave-b83v krb5kdc[24108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:23:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:23:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790626764250411-3144-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790626764250411-3144-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 20:23:44.391921 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790626764250411-3144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790626764250411-3144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790626764250411-3144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46313
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790626764250411-3144-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46313
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:23:44.474890 24121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:23:44.475224 24121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:23:44.475277 24121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:23:44.476747 24121 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:23:44.476837 24121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:23:44.476861 24121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:23:44.476881 24121 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:23:44.476922 24121 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:23:44.478667 24121 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790626764250411-3144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790626764250411-3144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790626764250411-3144-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790626764250411-3144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:23:44.478992 24121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:23:44.479252 24121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 20:23:44.484683 24121 server_base.cc:1034] running on GCE node
W20260928 20:23:44.484660 24129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:44.484664 24127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:44.484728 24126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:23:44.485200 24121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:23:44.485527 24121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:23:44.486676 24121 hybrid_clock.cc:648] HybridClock initialized: now 1790627024486647 us; error 40 us; skew 500 ppm
Sep 28 20:23:44 dist-test-slave-b83v krb5kdc[24108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627024, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:23:44.489621 24121 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 20:23:44.490228 24121 webserver.cc:460] Webserver started at http://127.0.0.1:42319/ using document root <none> and password file <none>
I20260928 20:23:44.490489 24121 fs_manager.cc:359] Metadata directory not provided
I20260928 20:23:44.490568 24121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:23:44.490726 24121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:23:44.492007 24121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790626764250411-3144-0/minicluster-data/master-0/data/instance:
uuid: "dfa96bbee3e64beb9394aed23678fb71"
format_stamp: "Formatted at 2026-09-28 20:23:44 on dist-test-slave-b83v"
server_key: "7620c3011a146b2568609bce3d475c81"
server_key_iv: "296bf87d13e6b4c59acbb222439aade8"
server_key_version: "encryptionkey@0"
I20260928 20:23:44.492445 24121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790626764250411-3144-0/minicluster-data/master-0/wal/instance:
uuid: "dfa96bbee3e64beb9394aed23678fb71"
format_stamp: "Formatted at 2026-09-28 20:23:44 on dist-test-slave-b83v"
server_key: "7620c3011a146b2568609bce3d475c81"
server_key_iv: "296bf87d13e6b4c59acbb222439aade8"
server_key_version: "encryptionkey@0"
I20260928 20:23:44.493973 24121 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 20:23:44.494846 24136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:44.495059 24121 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:23:44.495172 24121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790626764250411-3144-0/minicluster-data/master-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790626764250411-3144-0/minicluster-data/master-0/wal
uuid: "dfa96bbee3e64beb9394aed23678fb71"
format_stamp: "Formatted at 2026-09-28 20:23:44 on dist-test-slave-b83v"
server_key: "7620c3011a146b2568609bce3d475c81"
server_key_iv: "296bf87d13e6b4c59acbb222439aade8"
server_key_version: "encryptionkey@0"
I20260928 20:23:44.495253 24121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790626764250411-3144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790626764250411-3144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790626764250411-3144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:23:44.510592 24121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:23:44.513041 24121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:23:44.513238 24121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:23:44.518123 24121 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46313
I20260928 20:23:44.518214 24188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46313 every 8 connection(s)
I20260928 20:23:44.518666 24121 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
I20260928 20:23:44.519711 24189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:23:44.522324 24189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfa96bbee3e64beb9394aed23678fb71: Bootstrap starting.
I20260928 20:23:44.522954 24189 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dfa96bbee3e64beb9394aed23678fb71: Neither blocks nor log segments found. Creating new log.
I20260928 20:23:44.523263 24189 log.cc:824] T 00000000000000000000000000000000 P dfa96bbee3e64beb9394aed23678fb71: Log is configured to *not* fsync() on all Append() calls
I20260928 20:23:44.524048 24189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfa96bbee3e64beb9394aed23678fb71: No bootstrap required, opened a new log
I20260928 20:23:44.525763 24189 raft_consensus.cc:348] T 00000000000000000000000000000000 P dfa96bbee3e64beb9394aed23678fb71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfa96bbee3e64beb9394aed23678fb71" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46313 } }
I20260928 20:23:44.525934 24189 raft_consensus.cc:374] T 00000000000000000000000000000000 P dfa96bbee3e64beb9394aed23678fb71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:23:44.525983 24189 raft_consensus.cc:729] T 00000000000000000000000000000000 P dfa96bbee3e64beb9394aed23678fb71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa96bbee3e64beb9394aed23678fb71, State: Initialized, Role: FOLLOWER
I20260928 20:23:44.526141 24189 consensus_queue.cc:260] T 00000000000000000000000000000000 P dfa96bbee3e64beb9394aed23678fb71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfa96bbee3e64beb9394aed23678fb71" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46313 } }
I20260928 20:23:44.526232 24189 raft_consensus.cc:388] T 00000000000000000000000000000000 P dfa96bbee3e64beb9394aed23678fb71 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:23:44.526278 24189 raft_consensus.cc:482] T 00000000000000000000000000000000 P dfa96bbee3e64beb9394aed23678fb71 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:23:44.526328 24189 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dfa96bbee3e64beb9394aed23678fb71 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:23:44.526880 24189 raft_consensus.cc:504] T 00000000000000000000000000000000 P dfa96bbee3e64beb9394aed23678fb71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfa96bbee3e64beb9394aed23678fb71" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46313 } }
I20260928 20:23:44.527025 24189 leader_election.cc:302] T 00000000000000000000000000000000 P dfa96bbee3e64beb9394aed23678fb71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dfa96bbee3e64beb9394aed23678fb71; no voters:
I20260928 20:23:44.527217 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 24121
I20260928 20:23:44.527235 24189 leader_election.cc:288] T 00000000000000000000000000000000 P dfa96bbee3e64beb9394aed23678fb71 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:23:44.527264 24192 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dfa96bbee3e64beb9394aed23678fb71 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:23:44.527371 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790626764250411-3144-0/minicluster-data/master-0/wal/instance
I20260928 20:23:44.527384 24192 raft_consensus.cc:686] T 00000000000000000000000000000000 P dfa96bbee3e64beb9394aed23678fb71 [term 1 LEADER]: Becoming Leader. State: Replica: dfa96bbee3e64beb9394aed23678fb71, State: Running, Role: LEADER
I20260928 20:23:44.527459 24192 consensus_queue.cc:237] T 00000000000000000000000000000000 P dfa96bbee3e64beb9394aed23678fb71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfa96bbee3e64beb9394aed23678fb71" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46313 } }
I20260928 20:23:44.527515 24189 sys_catalog.cc:564] T 00000000000000000000000000000000 P dfa96bbee3e64beb9394aed23678fb71 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:23:44.527588 3144 external_mini_cluster.cc:1428] Setting key 5c0ae92b303e410f424ab1e4176d76ab
I20260928 20:23:44.527964 24193 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfa96bbee3e64beb9394aed23678fb71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dfa96bbee3e64beb9394aed23678fb71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfa96bbee3e64beb9394aed23678fb71" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46313 } } }
I20260928 20:23:44.528062 24193 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfa96bbee3e64beb9394aed23678fb71 [sys.catalog]: This master's current role is: LEADER
I20260928 20:23:44.528275 24194 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfa96bbee3e64beb9394aed23678fb71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dfa96bbee3e64beb9394aed23678fb71. Latest consensus state: current_term: 1 leader_uuid: "dfa96bbee3e64beb9394aed23678fb71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfa96bbee3e64beb9394aed23678fb71" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46313 } } }
I20260928 20:23:44.528348 24194 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfa96bbee3e64beb9394aed23678fb71 [sys.catalog]: This master's current role is: LEADER
I20260928 20:23:44.528465 24201 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:23:44.529084 24201 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:23:44.531049 24201 catalog_manager.cc:1347] Generated new cluster ID: fc25131a94dc4c8793a58dd7248578c0
I20260928 20:23:44.531121 24201 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:23:44.533898 24203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:44.528468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.165:52719 (local address 127.0.0.1:46313)
0928 20:23:44.528664 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:44.528676 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:44.528754 (+ 78us) server_negotiation.cc:407] Connection header received
0928 20:23:44.528930 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:44.528939 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:44.529022 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:44.529579 (+ 557us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:44.529691 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:44.530928 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:44.531863 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:44.532306 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:44.532451 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:44.533238 (+ 787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:44.533242 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:44.533243 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:23:44.533434 (+ 191us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:44.533465 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:23:44.533477 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:44.533576 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:44.533666 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 20:23:44.533705 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":89,"threads_started":1}
I20260928 20:23:44.534967 3144 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 20:23:44.542905 24203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:44.536687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.165:38433 (local address 127.0.0.1:46313)
0928 20:23:44.536745 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:44.536747 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:44.539989 (+ 3242us) server_negotiation.cc:407] Connection header received
0928 20:23:44.540072 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:44.540073 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:44.540101 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:44.540160 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:44.540308 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:44.540812 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:44.541566 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:44.541818 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:44.541901 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:44.542638 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:44.542640 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:44.542641 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:23:44.542645 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:44.542663 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:23:44.542668 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:44.542723 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:44.542783 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 20:23:44.542798 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 20:23:44.551385 24201 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:23:44.552105 24201 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:23:44.557123 24201 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dfa96bbee3e64beb9394aed23678fb71: Generated new TSK 0
I20260928 20:23:44.557397 24201 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 20:23:44.562939 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 24121
W20260928 20:23:44.567525 24106 connection.cc:535] client connection to 127.0.0.1:46313 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46313): Connection reset by peer (error 104)
2026-09-28T20:23:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1366 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790626764250411-3144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:23:44 dist-test-slave-b83v krb5kdc[24220](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:23:44 dist-test-slave-b83v krb5kdc[24220](info): set up 2 sockets
Sep 28 20:23:44 dist-test-slave-b83v krb5kdc[24220](info): commencing operation
krb5kdc: starting...
W20260928 20:23:45.624452 3144 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Sep 28 20:23:45 dist-test-slave-b83v krb5kdc[24220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:23:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:23:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790626764250411-3144-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790626764250411-3144-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 20:23:45.751324 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790626764250411-3144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790626764250411-3144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790626764250411-3144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43595
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790626764250411-3144-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:23:45.833806 24233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:23:45.834045 24233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:23:45.834071 24233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:23:45.835342 24233 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:23:45.835395 24233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:23:45.835410 24233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:23:45.835426 24233 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:23:45.835438 24233 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:23:45.837112 24233 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790626764250411-3144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790626764250411-3144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790626764250411-3144-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790626764250411-3144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:23:45.837393 24233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:23:45.837662 24233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 20:23:45.842947 24233 server_base.cc:1034] running on GCE node
W20260928 20:23:45.842914 24239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:45.842914 24238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:45.842904 24241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:23:45.843425 24233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:23:45.843854 24233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:23:45.845024 24233 hybrid_clock.cc:648] HybridClock initialized: now 1790627025844989 us; error 49 us; skew 500 ppm
Sep 28 20:23:45 dist-test-slave-b83v krb5kdc[24220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627025, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:23:45.847995 24233 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 20:23:45.848479 24233 webserver.cc:460] Webserver started at http://127.0.0.1:39869/ using document root <none> and password file <none>
I20260928 20:23:45.848742 24233 fs_manager.cc:359] Metadata directory not provided
I20260928 20:23:45.848816 24233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:23:45.848971 24233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:23:45.850155 24233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790626764250411-3144-0/minicluster-data/master-0/data/instance:
uuid: "b80a865af9884cd092bd51b8df36f554"
format_stamp: "Formatted at 2026-09-28 20:23:45 on dist-test-slave-b83v"
server_key: "1525e7012ebe602b4a8e75a39193fbc3"
server_key_iv: "c0c32bd2655890260b435b47f52314d2"
server_key_version: "encryptionkey@0"
I20260928 20:23:45.850538 24233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790626764250411-3144-0/minicluster-data/master-0/wal/instance:
uuid: "b80a865af9884cd092bd51b8df36f554"
format_stamp: "Formatted at 2026-09-28 20:23:45 on dist-test-slave-b83v"
server_key: "1525e7012ebe602b4a8e75a39193fbc3"
server_key_iv: "c0c32bd2655890260b435b47f52314d2"
server_key_version: "encryptionkey@0"
I20260928 20:23:45.852066 24233 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:23:45.852974 24248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:45.853282 24233 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:23:45.853418 24233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790626764250411-3144-0/minicluster-data/master-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790626764250411-3144-0/minicluster-data/master-0/wal
uuid: "b80a865af9884cd092bd51b8df36f554"
format_stamp: "Formatted at 2026-09-28 20:23:45 on dist-test-slave-b83v"
server_key: "1525e7012ebe602b4a8e75a39193fbc3"
server_key_iv: "c0c32bd2655890260b435b47f52314d2"
server_key_version: "encryptionkey@0"
I20260928 20:23:45.853504 24233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790626764250411-3144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790626764250411-3144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790626764250411-3144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:23:45.857510 24233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:23:45.859894 24233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:23:45.860090 24233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:23:45.864795 24233 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43595
I20260928 20:23:45.864874 24300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43595 every 8 connection(s)
I20260928 20:23:45.865285 24233 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
I20260928 20:23:45.866400 24301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:23:45.866511 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 24233
I20260928 20:23:45.866648 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790626764250411-3144-0/minicluster-data/master-0/wal/instance
I20260928 20:23:45.866861 3144 external_mini_cluster.cc:1428] Setting key 3f0fcd2b04944a0160a45f89bbb9d1e9
I20260928 20:23:45.869859 24304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:45.867436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.165:40591 (local address 127.0.0.1:43595)
0928 20:23:45.867650 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:45.867662 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:45.868039 (+ 377us) server_negotiation.cc:407] Connection header received
0928 20:23:45.868495 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:45.868502 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:45.868532 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:45.868662 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:45.869398 (+ 736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:45.869402 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:45.869402 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:23:45.869406 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:45.869445 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:23:45.869455 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:45.869512 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:45.869606 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 20:23:45.869670 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":91,"threads_started":1}
I20260928 20:23:45.870146 24301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b80a865af9884cd092bd51b8df36f554: Bootstrap starting.
I20260928 20:23:45.870918 24301 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b80a865af9884cd092bd51b8df36f554: Neither blocks nor log segments found. Creating new log.
I20260928 20:23:45.871019 3144 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 20:23:45.871241 24301 log.cc:824] T 00000000000000000000000000000000 P b80a865af9884cd092bd51b8df36f554: Log is configured to *not* fsync() on all Append() calls
I20260928 20:23:45.872053 24301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b80a865af9884cd092bd51b8df36f554: No bootstrap required, opened a new log
I20260928 20:23:45.873661 24304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:45.872605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.165:59249 (local address 127.0.0.1:43595)
0928 20:23:45.872647 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:45.872648 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:45.872721 (+ 73us) server_negotiation.cc:407] Connection header received
0928 20:23:45.872763 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:45.872764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:45.872769 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:45.872839 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:45.873445 (+ 606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:45.873447 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:45.873447 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:23:45.873451 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:45.873467 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:23:45.873472 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:45.873547 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:45.873581 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 20:23:45.873594 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 20:23:45.874320 24301 raft_consensus.cc:348] T 00000000000000000000000000000000 P b80a865af9884cd092bd51b8df36f554 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b80a865af9884cd092bd51b8df36f554" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43595 } }
I20260928 20:23:45.874457 24301 raft_consensus.cc:374] T 00000000000000000000000000000000 P b80a865af9884cd092bd51b8df36f554 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:23:45.874480 24301 raft_consensus.cc:729] T 00000000000000000000000000000000 P b80a865af9884cd092bd51b8df36f554 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b80a865af9884cd092bd51b8df36f554, State: Initialized, Role: FOLLOWER
I20260928 20:23:45.874603 24301 consensus_queue.cc:260] T 00000000000000000000000000000000 P b80a865af9884cd092bd51b8df36f554 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b80a865af9884cd092bd51b8df36f554" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43595 } }
I20260928 20:23:45.874682 24301 raft_consensus.cc:388] T 00000000000000000000000000000000 P b80a865af9884cd092bd51b8df36f554 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:23:45.874706 24301 raft_consensus.cc:482] T 00000000000000000000000000000000 P b80a865af9884cd092bd51b8df36f554 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:23:45.874738 24301 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b80a865af9884cd092bd51b8df36f554 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:23:45.875249 24301 raft_consensus.cc:504] T 00000000000000000000000000000000 P b80a865af9884cd092bd51b8df36f554 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b80a865af9884cd092bd51b8df36f554" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43595 } }
I20260928 20:23:45.875335 24301 leader_election.cc:302] T 00000000000000000000000000000000 P b80a865af9884cd092bd51b8df36f554 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b80a865af9884cd092bd51b8df36f554; no voters:
I20260928 20:23:45.875460 24301 leader_election.cc:288] T 00000000000000000000000000000000 P b80a865af9884cd092bd51b8df36f554 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:23:45.875528 24311 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b80a865af9884cd092bd51b8df36f554 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:23:45.875738 24301 sys_catalog.cc:564] T 00000000000000000000000000000000 P b80a865af9884cd092bd51b8df36f554 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:23:45.875797 24311 raft_consensus.cc:686] T 00000000000000000000000000000000 P b80a865af9884cd092bd51b8df36f554 [term 1 LEADER]: Becoming Leader. State: Replica: b80a865af9884cd092bd51b8df36f554, State: Running, Role: LEADER
I20260928 20:23:45.875952 24311 consensus_queue.cc:237] T 00000000000000000000000000000000 P b80a865af9884cd092bd51b8df36f554 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b80a865af9884cd092bd51b8df36f554" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43595 } }
I20260928 20:23:45.876395 24312 sys_catalog.cc:455] T 00000000000000000000000000000000 P b80a865af9884cd092bd51b8df36f554 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b80a865af9884cd092bd51b8df36f554" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b80a865af9884cd092bd51b8df36f554" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43595 } } }
I20260928 20:23:45.876537 24312 sys_catalog.cc:458] T 00000000000000000000000000000000 P b80a865af9884cd092bd51b8df36f554 [sys.catalog]: This master's current role is: LEADER
I20260928 20:23:45.876767 24313 sys_catalog.cc:455] T 00000000000000000000000000000000 P b80a865af9884cd092bd51b8df36f554 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b80a865af9884cd092bd51b8df36f554. Latest consensus state: current_term: 1 leader_uuid: "b80a865af9884cd092bd51b8df36f554" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b80a865af9884cd092bd51b8df36f554" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43595 } } }
I20260928 20:23:45.876873 24313 sys_catalog.cc:458] T 00000000000000000000000000000000 P b80a865af9884cd092bd51b8df36f554 [sys.catalog]: This master's current role is: LEADER
I20260928 20:23:45.877054 24320 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:23:45.877564 24320 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:23:45.879244 24320 catalog_manager.cc:1347] Generated new cluster ID: f8473d548bfd4b69b46e4b42a1785c4e
I20260928 20:23:45.879300 24320 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:23:45.887238 24320 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:23:45.887463 24320 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:23:45.894155 24320 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b80a865af9884cd092bd51b8df36f554: Generated new TSK 0
I20260928 20:23:45.894356 24320 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 20:23:45.902825 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 24233
2026-09-28T20:23:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1339 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790626764250411-3144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:23:45 dist-test-slave-b83v krb5kdc[24332](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:23:45 dist-test-slave-b83v krb5kdc[24332](info): set up 2 sockets
Sep 28 20:23:45 dist-test-slave-b83v krb5kdc[24332](info): commencing operation
krb5kdc: starting...
W20260928 20:23:46.932783 3144 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Sep 28 20:23:47 dist-test-slave-b83v krb5kdc[24332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:23:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:23:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790626764250411-3144-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790626764250411-3144-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 20:23:47.059803 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790626764250411-3144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790626764250411-3144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790626764250411-3144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41271
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790626764250411-3144-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41271
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:23:47.146384 24345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:23:47.146656 24345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:23:47.146687 24345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:23:47.147957 24345 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:23:47.148000 24345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:23:47.148015 24345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:23:47.148027 24345 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:23:47.148039 24345 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:23:47.149646 24345 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790626764250411-3144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790626764250411-3144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41271
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790626764250411-3144-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24345
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790626764250411-3144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:23:47.149861 24345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:23:47.150049 24345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:23:47.155372 24350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:47.155347 24353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:47.155386 24351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:23:47.155812 24345 server_base.cc:1034] running on GCE node
I20260928 20:23:47.156301 24345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:23:47.156653 24345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:23:47.157862 24345 hybrid_clock.cc:648] HybridClock initialized: now 1790627027157794 us; error 91 us; skew 500 ppm
Sep 28 20:23:47 dist-test-slave-b83v krb5kdc[24332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:23:47.160882 24345 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 20:23:47.161473 24345 webserver.cc:460] Webserver started at http://127.0.0.1:36887/ using document root <none> and password file <none>
I20260928 20:23:47.161727 24345 fs_manager.cc:359] Metadata directory not provided
I20260928 20:23:47.161798 24345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:23:47.161950 24345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:23:47.163053 24345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790626764250411-3144-0/minicluster-data/master-0/data/instance:
uuid: "ae5c9c06c1ff4e02859531fb81f9d10a"
format_stamp: "Formatted at 2026-09-28 20:23:47 on dist-test-slave-b83v"
server_key: "ddd236581eb0f5c6dec73697e9198a69"
server_key_iv: "bfb0d5f48798955c8d2caa43b83446fd"
server_key_version: "encryptionkey@0"
I20260928 20:23:47.163441 24345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790626764250411-3144-0/minicluster-data/master-0/wal/instance:
uuid: "ae5c9c06c1ff4e02859531fb81f9d10a"
format_stamp: "Formatted at 2026-09-28 20:23:47 on dist-test-slave-b83v"
server_key: "ddd236581eb0f5c6dec73697e9198a69"
server_key_iv: "bfb0d5f48798955c8d2caa43b83446fd"
server_key_version: "encryptionkey@0"
I20260928 20:23:47.164983 24345 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:23:47.165915 24360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:47.166193 24345 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:23:47.166342 24345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790626764250411-3144-0/minicluster-data/master-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790626764250411-3144-0/minicluster-data/master-0/wal
uuid: "ae5c9c06c1ff4e02859531fb81f9d10a"
format_stamp: "Formatted at 2026-09-28 20:23:47 on dist-test-slave-b83v"
server_key: "ddd236581eb0f5c6dec73697e9198a69"
server_key_iv: "bfb0d5f48798955c8d2caa43b83446fd"
server_key_version: "encryptionkey@0"
I20260928 20:23:47.166430 24345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790626764250411-3144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790626764250411-3144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790626764250411-3144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:23:47.183162 24345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:23:47.186192 24345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:23:47.186431 24345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:23:47.191888 24345 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41271
I20260928 20:23:47.191972 24412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41271 every 8 connection(s)
I20260928 20:23:47.192361 24345 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
I20260928 20:23:47.193500 24413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:23:47.195493 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 24345
I20260928 20:23:47.195663 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790626764250411-3144-0/minicluster-data/master-0/wal/instance
I20260928 20:23:47.195902 3144 external_mini_cluster.cc:1428] Setting key f7f81c72349adfecf4ed1cbdc333a043
I20260928 20:23:47.196192 24413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae5c9c06c1ff4e02859531fb81f9d10a: Bootstrap starting.
I20260928 20:23:47.196997 24413 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae5c9c06c1ff4e02859531fb81f9d10a: Neither blocks nor log segments found. Creating new log.
I20260928 20:23:47.197315 24413 log.cc:824] T 00000000000000000000000000000000 P ae5c9c06c1ff4e02859531fb81f9d10a: Log is configured to *not* fsync() on all Append() calls
I20260928 20:23:47.198220 24413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae5c9c06c1ff4e02859531fb81f9d10a: No bootstrap required, opened a new log
I20260928 20:23:47.200625 24413 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae5c9c06c1ff4e02859531fb81f9d10a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae5c9c06c1ff4e02859531fb81f9d10a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41271 } }
I20260928 20:23:47.200837 24413 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae5c9c06c1ff4e02859531fb81f9d10a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:23:47.200863 24413 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae5c9c06c1ff4e02859531fb81f9d10a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae5c9c06c1ff4e02859531fb81f9d10a, State: Initialized, Role: FOLLOWER
I20260928 20:23:47.201023 24413 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae5c9c06c1ff4e02859531fb81f9d10a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae5c9c06c1ff4e02859531fb81f9d10a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41271 } }
I20260928 20:23:47.201088 24413 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae5c9c06c1ff4e02859531fb81f9d10a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:23:47.201131 24413 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae5c9c06c1ff4e02859531fb81f9d10a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:23:47.201176 24413 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae5c9c06c1ff4e02859531fb81f9d10a [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:23:47.202045 24413 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae5c9c06c1ff4e02859531fb81f9d10a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae5c9c06c1ff4e02859531fb81f9d10a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41271 } }
I20260928 20:23:47.202193 24413 leader_election.cc:302] T 00000000000000000000000000000000 P ae5c9c06c1ff4e02859531fb81f9d10a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae5c9c06c1ff4e02859531fb81f9d10a; no voters:
I20260928 20:23:47.202399 24413 leader_election.cc:288] T 00000000000000000000000000000000 P ae5c9c06c1ff4e02859531fb81f9d10a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:23:47.202493 24418 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae5c9c06c1ff4e02859531fb81f9d10a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:23:47.202746 24413 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae5c9c06c1ff4e02859531fb81f9d10a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:23:47.202818 24418 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae5c9c06c1ff4e02859531fb81f9d10a [term 1 LEADER]: Becoming Leader. State: Replica: ae5c9c06c1ff4e02859531fb81f9d10a, State: Running, Role: LEADER
I20260928 20:23:47.202955 24418 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae5c9c06c1ff4e02859531fb81f9d10a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae5c9c06c1ff4e02859531fb81f9d10a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41271 } }
I20260928 20:23:47.203380 24419 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae5c9c06c1ff4e02859531fb81f9d10a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae5c9c06c1ff4e02859531fb81f9d10a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae5c9c06c1ff4e02859531fb81f9d10a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41271 } } }
I20260928 20:23:47.203493 24419 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae5c9c06c1ff4e02859531fb81f9d10a [sys.catalog]: This master's current role is: LEADER
I20260928 20:23:47.203372 24418 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae5c9c06c1ff4e02859531fb81f9d10a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae5c9c06c1ff4e02859531fb81f9d10a. Latest consensus state: current_term: 1 leader_uuid: "ae5c9c06c1ff4e02859531fb81f9d10a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae5c9c06c1ff4e02859531fb81f9d10a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41271 } } }
I20260928 20:23:47.203740 24418 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae5c9c06c1ff4e02859531fb81f9d10a [sys.catalog]: This master's current role is: LEADER
Sep 28 20:23:47 dist-test-slave-b83v krb5kdc[24332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 20:23:47.204294 24427 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:23:47.205534 24427 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:23:47.207490 24427 catalog_manager.cc:1347] Generated new cluster ID: 459137693d284516b3f0cb07681083cd
I20260928 20:23:47.207582 24427 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:23:47.209717 24416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:47.196498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.165:38563 (local address 127.0.0.1:41271)
0928 20:23:47.196730 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:47.196739 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:47.196769 (+ 30us) server_negotiation.cc:407] Connection header received
0928 20:23:47.197299 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:47.197307 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:47.197364 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:47.197526 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:47.198374 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:47.199429 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:47.200715 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:47.201367 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:47.201471 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:47.205142 (+ 3671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:47.205147 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:47.205148 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:23:47.205169 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:47.207475 (+ 2306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:47.208103 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:47.208108 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:47.208108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:47.208247 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:47.208530 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:47.208531 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:47.208531 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:47.208905 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:47.209083 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:47.209357 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 20:23:47.209475 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":85,"threads_started":1}
I20260928 20:23:47.210773 3144 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 20:23:47.220218 24416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:47.212340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.165:32837 (local address 127.0.0.1:41271)
0928 20:23:47.212422 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:47.212424 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:47.212573 (+ 149us) server_negotiation.cc:407] Connection header received
0928 20:23:47.212606 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:47.212607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:47.212619 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:47.212688 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:47.213353 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:47.214162 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:47.215174 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:47.215536 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:47.215624 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:47.217104 (+ 1480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:47.217106 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:47.217106 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:23:47.217114 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:47.218787 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:47.219247 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:47.219248 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:47.219248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:47.219314 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:47.219549 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:47.219550 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:47.219551 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:47.219706 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:47.219855 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:47.220025 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 20:23:47.220104 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 20:23:47.221343 24427 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:23:47.221907 24427 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:23:47.225869 24427 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae5c9c06c1ff4e02859531fb81f9d10a: Generated new TSK 0
I20260928 20:23:47.226121 24427 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 20:23:47.229403 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 24345
W20260928 20:23:47.234206 24330 connection.cc:535] client connection to 127.0.0.1:41271 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41271): Connection reset by peer (error 104)
2026-09-28T20:23:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1327 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790626764250411-3144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:23:47 dist-test-slave-b83v krb5kdc[24444](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:23:47 dist-test-slave-b83v krb5kdc[24444](info): set up 2 sockets
Sep 28 20:23:47 dist-test-slave-b83v krb5kdc[24444](info): commencing operation
krb5kdc: starting...
W20260928 20:23:48.296849 3144 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Sep 28 20:23:48 dist-test-slave-b83v krb5kdc[24444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:23:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:23:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790626764250411-3144-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790626764250411-3144-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 20:23:48.423189 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790626764250411-3144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790626764250411-3144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790626764250411-3144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43701
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790626764250411-3144-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43701
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:23:48.509442 24457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:23:48.509670 24457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:23:48.509694 24457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:23:48.510823 24457 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:23:48.510865 24457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:23:48.510879 24457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:23:48.510891 24457 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:23:48.510903 24457 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:23:48.512465 24457 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790626764250411-3144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790626764250411-3144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43701
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790626764250411-3144-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790626764250411-3144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:23:48.512678 24457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:23:48.512848 24457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:23:48.518015 24465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:48.518021 24462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:48.518172 24463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:23:48.518044 24457 server_base.cc:1034] running on GCE node
I20260928 20:23:48.518487 24457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:23:48.518808 24457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:23:48.519956 24457 hybrid_clock.cc:648] HybridClock initialized: now 1790627028519913 us; error 54 us; skew 500 ppm
Sep 28 20:23:48 dist-test-slave-b83v krb5kdc[24444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627028, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:23:48.526670 24457 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 20:23:48.527184 24457 webserver.cc:460] Webserver started at http://127.0.0.1:39265/ using document root <none> and password file <none>
I20260928 20:23:48.527449 24457 fs_manager.cc:359] Metadata directory not provided
I20260928 20:23:48.527522 24457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:23:48.527689 24457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:23:48.528990 24457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790626764250411-3144-0/minicluster-data/master-0/data/instance:
uuid: "f78f7dd997a14370995f51d848b2960f"
format_stamp: "Formatted at 2026-09-28 20:23:48 on dist-test-slave-b83v"
server_key: "e6b6b08729d4fe6f405162ea7a288b82"
server_key_iv: "e332063ca08f567336edeab40180300e"
server_key_version: "encryptionkey@0"
I20260928 20:23:48.529385 24457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790626764250411-3144-0/minicluster-data/master-0/wal/instance:
uuid: "f78f7dd997a14370995f51d848b2960f"
format_stamp: "Formatted at 2026-09-28 20:23:48 on dist-test-slave-b83v"
server_key: "e6b6b08729d4fe6f405162ea7a288b82"
server_key_iv: "e332063ca08f567336edeab40180300e"
server_key_version: "encryptionkey@0"
I20260928 20:23:48.530810 24457 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:23:48.531683 24472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:48.532049 24457 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:23:48.532171 24457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790626764250411-3144-0/minicluster-data/master-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790626764250411-3144-0/minicluster-data/master-0/wal
uuid: "f78f7dd997a14370995f51d848b2960f"
format_stamp: "Formatted at 2026-09-28 20:23:48 on dist-test-slave-b83v"
server_key: "e6b6b08729d4fe6f405162ea7a288b82"
server_key_iv: "e332063ca08f567336edeab40180300e"
server_key_version: "encryptionkey@0"
I20260928 20:23:48.532259 24457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790626764250411-3144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790626764250411-3144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790626764250411-3144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:23:48.543725 24457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:23:48.546136 24457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:23:48.546324 24457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:23:48.550922 24457 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43701
I20260928 20:23:48.551052 24524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43701 every 8 connection(s)
I20260928 20:23:48.551380 24457 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
I20260928 20:23:48.552372 24525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:23:48.554836 24525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f78f7dd997a14370995f51d848b2960f: Bootstrap starting.
I20260928 20:23:48.555441 24525 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f78f7dd997a14370995f51d848b2960f: Neither blocks nor log segments found. Creating new log.
I20260928 20:23:48.555751 24525 log.cc:824] T 00000000000000000000000000000000 P f78f7dd997a14370995f51d848b2960f: Log is configured to *not* fsync() on all Append() calls
I20260928 20:23:48.556710 24525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f78f7dd997a14370995f51d848b2960f: No bootstrap required, opened a new log
I20260928 20:23:48.558463 24525 raft_consensus.cc:348] T 00000000000000000000000000000000 P f78f7dd997a14370995f51d848b2960f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f78f7dd997a14370995f51d848b2960f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43701 } }
I20260928 20:23:48.558642 24525 raft_consensus.cc:374] T 00000000000000000000000000000000 P f78f7dd997a14370995f51d848b2960f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:23:48.558694 24525 raft_consensus.cc:729] T 00000000000000000000000000000000 P f78f7dd997a14370995f51d848b2960f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f78f7dd997a14370995f51d848b2960f, State: Initialized, Role: FOLLOWER
I20260928 20:23:48.558867 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 24457
I20260928 20:23:48.558869 24525 consensus_queue.cc:260] T 00000000000000000000000000000000 P f78f7dd997a14370995f51d848b2960f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f78f7dd997a14370995f51d848b2960f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43701 } }
I20260928 20:23:48.558959 24525 raft_consensus.cc:388] T 00000000000000000000000000000000 P f78f7dd997a14370995f51d848b2960f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:23:48.558991 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790626764250411-3144-0/minicluster-data/master-0/wal/instance
I20260928 20:23:48.559007 24525 raft_consensus.cc:482] T 00000000000000000000000000000000 P f78f7dd997a14370995f51d848b2960f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:23:48.559065 24525 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f78f7dd997a14370995f51d848b2960f [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:23:48.559214 3144 external_mini_cluster.cc:1428] Setting key cc9c9aad03fed4456a7b48c05002a1a8
I20260928 20:23:48.559877 24525 raft_consensus.cc:504] T 00000000000000000000000000000000 P f78f7dd997a14370995f51d848b2960f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f78f7dd997a14370995f51d848b2960f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43701 } }
I20260928 20:23:48.560014 24525 leader_election.cc:302] T 00000000000000000000000000000000 P f78f7dd997a14370995f51d848b2960f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f78f7dd997a14370995f51d848b2960f; no voters:
I20260928 20:23:48.560180 24525 leader_election.cc:288] T 00000000000000000000000000000000 P f78f7dd997a14370995f51d848b2960f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:23:48.560209 24530 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f78f7dd997a14370995f51d848b2960f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:23:48.560315 24530 raft_consensus.cc:686] T 00000000000000000000000000000000 P f78f7dd997a14370995f51d848b2960f [term 1 LEADER]: Becoming Leader. State: Replica: f78f7dd997a14370995f51d848b2960f, State: Running, Role: LEADER
I20260928 20:23:48.560467 24525 sys_catalog.cc:564] T 00000000000000000000000000000000 P f78f7dd997a14370995f51d848b2960f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:23:48.560477 24530 consensus_queue.cc:237] T 00000000000000000000000000000000 P f78f7dd997a14370995f51d848b2960f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f78f7dd997a14370995f51d848b2960f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43701 } }
I20260928 20:23:48.561038 24531 sys_catalog.cc:455] T 00000000000000000000000000000000 P f78f7dd997a14370995f51d848b2960f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f78f7dd997a14370995f51d848b2960f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f78f7dd997a14370995f51d848b2960f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43701 } } }
I20260928 20:23:48.561094 24532 sys_catalog.cc:455] T 00000000000000000000000000000000 P f78f7dd997a14370995f51d848b2960f [sys.catalog]: SysCatalogTable state changed. Reason: New leader f78f7dd997a14370995f51d848b2960f. Latest consensus state: current_term: 1 leader_uuid: "f78f7dd997a14370995f51d848b2960f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f78f7dd997a14370995f51d848b2960f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43701 } } }
I20260928 20:23:48.561259 24531 sys_catalog.cc:458] T 00000000000000000000000000000000 P f78f7dd997a14370995f51d848b2960f [sys.catalog]: This master's current role is: LEADER
I20260928 20:23:48.561273 24532 sys_catalog.cc:458] T 00000000000000000000000000000000 P f78f7dd997a14370995f51d848b2960f [sys.catalog]: This master's current role is: LEADER
I20260928 20:23:48.561759 24538 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:23:48.562376 24538 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:23:48.564159 24538 catalog_manager.cc:1347] Generated new cluster ID: 8718e30ff501489aa9070cffe804c24b
I20260928 20:23:48.564216 24538 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:23:48.569941 24529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:48.559733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.165:38609 (local address 127.0.0.1:43701)
0928 20:23:48.559948 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:48.559962 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:48.559999 (+ 37us) server_negotiation.cc:407] Connection header received
0928 20:23:48.560138 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:48.560145 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:48.560211 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:48.560753 (+ 542us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:48.560838 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:48.562651 (+ 1813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:48.563832 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:48.568198 (+ 4366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:48.568331 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:48.569182 (+ 851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:48.569186 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:48.569187 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:23:48.569391 (+ 204us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:48.569430 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:23:48.569444 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:48.569525 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:48.569681 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 20:23:48.569713 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":113,"threads_started":1}
I20260928 20:23:48.570993 3144 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 20:23:48.576152 24529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:48.572387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.165:59121 (local address 127.0.0.1:43701)
0928 20:23:48.572427 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:48.572429 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:48.572631 (+ 202us) server_negotiation.cc:407] Connection header received
0928 20:23:48.572688 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:48.572689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:48.572700 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:48.572757 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:48.572893 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:48.573639 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:48.574675 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:48.575025 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:48.575100 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:48.575895 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:48.575896 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:48.575897 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:23:48.575902 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:48.575916 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:23:48.575921 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:48.575981 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:48.576061 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 20:23:48.576073 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 20:23:48.581250 24538 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:23:48.581935 24538 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:23:48.587525 24538 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f78f7dd997a14370995f51d848b2960f: Generated new TSK 0
I20260928 20:23:48.587791 24538 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 20:23:48.592742 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 24457
W20260928 20:23:48.597230 24442 connection.cc:535] client connection to 127.0.0.1:43701 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43701): Connection reset by peer (error 104)
2026-09-28T20:23:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1363 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790626764250411-3144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:23:48 dist-test-slave-b83v krb5kdc[24556](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:23:48 dist-test-slave-b83v krb5kdc[24556](info): set up 2 sockets
Sep 28 20:23:48 dist-test-slave-b83v krb5kdc[24556](info): commencing operation
krb5kdc: starting...
W20260928 20:23:49.668246 3144 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Sep 28 20:23:49 dist-test-slave-b83v krb5kdc[24556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:23:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:23:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790626764250411-3144-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790626764250411-3144-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 20:23:49.796041 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790626764250411-3144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790626764250411-3144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790626764250411-3144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34323
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790626764250411-3144-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34323
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:23:49.886600 24569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:23:49.886902 24569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:23:49.886950 24569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:23:49.888322 24569 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:23:49.888412 24569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:23:49.888441 24569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:23:49.888473 24569 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:23:49.888506 24569 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:23:49.890130 24569 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790626764250411-3144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790626764250411-3144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34323
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790626764250411-3144-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790626764250411-3144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:23:49.890462 24569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:23:49.891036 24569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:23:49.896589 24575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:49.896632 24577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:49.896906 24574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:23:49.896912 24569 server_base.cc:1034] running on GCE node
I20260928 20:23:49.897251 24569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:23:49.897567 24569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:23:49.898752 24569 hybrid_clock.cc:648] HybridClock initialized: now 1790627029898732 us; error 66 us; skew 500 ppm
Sep 28 20:23:49 dist-test-slave-b83v krb5kdc[24556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627029, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:23:49.902010 24569 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 20:23:49.902493 24569 webserver.cc:460] Webserver started at http://127.0.0.1:43283/ using document root <none> and password file <none>
I20260928 20:23:49.902771 24569 fs_manager.cc:359] Metadata directory not provided
I20260928 20:23:49.902822 24569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:23:49.903014 24569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:23:49.904232 24569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790626764250411-3144-0/minicluster-data/master-0/data/instance:
uuid: "e849daa0672d4805ba5cffa3d44b6d8f"
format_stamp: "Formatted at 2026-09-28 20:23:49 on dist-test-slave-b83v"
server_key: "ecfc046097f71a5e6951847e605cca1c"
server_key_iv: "062bca4981d6af09cc432e5d6bf40739"
server_key_version: "encryptionkey@0"
I20260928 20:23:49.904645 24569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790626764250411-3144-0/minicluster-data/master-0/wal/instance:
uuid: "e849daa0672d4805ba5cffa3d44b6d8f"
format_stamp: "Formatted at 2026-09-28 20:23:49 on dist-test-slave-b83v"
server_key: "ecfc046097f71a5e6951847e605cca1c"
server_key_iv: "062bca4981d6af09cc432e5d6bf40739"
server_key_version: "encryptionkey@0"
I20260928 20:23:49.906454 24569 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 20:23:49.907632 24584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:49.907915 24569 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:23:49.908032 24569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790626764250411-3144-0/minicluster-data/master-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790626764250411-3144-0/minicluster-data/master-0/wal
uuid: "e849daa0672d4805ba5cffa3d44b6d8f"
format_stamp: "Formatted at 2026-09-28 20:23:49 on dist-test-slave-b83v"
server_key: "ecfc046097f71a5e6951847e605cca1c"
server_key_iv: "062bca4981d6af09cc432e5d6bf40739"
server_key_version: "encryptionkey@0"
I20260928 20:23:49.908126 24569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790626764250411-3144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790626764250411-3144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790626764250411-3144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:23:49.912375 24569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:23:49.914901 24569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:23:49.915086 24569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:23:49.919361 24569 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34323
I20260928 20:23:49.919401 24636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34323 every 8 connection(s)
I20260928 20:23:49.919854 24569 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
I20260928 20:23:49.921023 24637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:23:49.921449 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 24569
I20260928 20:23:49.921613 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790626764250411-3144-0/minicluster-data/master-0/wal/instance
I20260928 20:23:49.921862 3144 external_mini_cluster.cc:1428] Setting key c6d62e4abddd3074437bae544a76e036
I20260928 20:23:49.924396 24637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e849daa0672d4805ba5cffa3d44b6d8f: Bootstrap starting.
I20260928 20:23:49.924693 24640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:49.922471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.165:47729 (local address 127.0.0.1:34323)
0928 20:23:49.922691 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:49.922702 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:49.922740 (+ 38us) server_negotiation.cc:407] Connection header received
0928 20:23:49.923291 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:49.923300 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:49.923338 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:49.923500 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:49.924261 (+ 761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:49.924265 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:49.924265 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:23:49.924269 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:49.924314 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:23:49.924328 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:49.924406 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:49.924462 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 20:23:49.924506 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":102,"threads_started":1}
I20260928 20:23:49.925207 24637 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e849daa0672d4805ba5cffa3d44b6d8f: Neither blocks nor log segments found. Creating new log.
I20260928 20:23:49.925527 24637 log.cc:824] T 00000000000000000000000000000000 P e849daa0672d4805ba5cffa3d44b6d8f: Log is configured to *not* fsync() on all Append() calls
I20260928 20:23:49.925793 3144 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 20:23:49.926306 24637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e849daa0672d4805ba5cffa3d44b6d8f: No bootstrap required, opened a new log
I20260928 20:23:49.928704 24640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:49.927455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.165:56745 (local address 127.0.0.1:34323)
0928 20:23:49.927497 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:49.927499 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:49.927561 (+ 62us) server_negotiation.cc:407] Connection header received
0928 20:23:49.927621 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:49.927622 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:49.927626 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:49.927723 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:49.928484 (+ 761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:49.928486 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:49.928487 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:23:49.928492 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:49.928512 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:23:49.928518 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:49.928585 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:49.928630 (+ 45us) server_negotiation.cc:299] Negotiation successful
0928 20:23:49.928644 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 20:23:49.928871 24637 raft_consensus.cc:348] T 00000000000000000000000000000000 P e849daa0672d4805ba5cffa3d44b6d8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e849daa0672d4805ba5cffa3d44b6d8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34323 } }
I20260928 20:23:49.929050 24637 raft_consensus.cc:374] T 00000000000000000000000000000000 P e849daa0672d4805ba5cffa3d44b6d8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:23:49.929096 24637 raft_consensus.cc:729] T 00000000000000000000000000000000 P e849daa0672d4805ba5cffa3d44b6d8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e849daa0672d4805ba5cffa3d44b6d8f, State: Initialized, Role: FOLLOWER
I20260928 20:23:49.929240 24637 consensus_queue.cc:260] T 00000000000000000000000000000000 P e849daa0672d4805ba5cffa3d44b6d8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e849daa0672d4805ba5cffa3d44b6d8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34323 } }
I20260928 20:23:49.929333 24637 raft_consensus.cc:388] T 00000000000000000000000000000000 P e849daa0672d4805ba5cffa3d44b6d8f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:23:49.929358 24637 raft_consensus.cc:482] T 00000000000000000000000000000000 P e849daa0672d4805ba5cffa3d44b6d8f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:23:49.929386 24637 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e849daa0672d4805ba5cffa3d44b6d8f [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:23:49.929941 24637 raft_consensus.cc:504] T 00000000000000000000000000000000 P e849daa0672d4805ba5cffa3d44b6d8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e849daa0672d4805ba5cffa3d44b6d8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34323 } }
I20260928 20:23:49.930032 24637 leader_election.cc:302] T 00000000000000000000000000000000 P e849daa0672d4805ba5cffa3d44b6d8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e849daa0672d4805ba5cffa3d44b6d8f; no voters:
I20260928 20:23:49.930162 24637 leader_election.cc:288] T 00000000000000000000000000000000 P e849daa0672d4805ba5cffa3d44b6d8f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:23:49.930243 24647 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e849daa0672d4805ba5cffa3d44b6d8f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:23:49.930454 24637 sys_catalog.cc:564] T 00000000000000000000000000000000 P e849daa0672d4805ba5cffa3d44b6d8f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:23:49.930544 24647 raft_consensus.cc:686] T 00000000000000000000000000000000 P e849daa0672d4805ba5cffa3d44b6d8f [term 1 LEADER]: Becoming Leader. State: Replica: e849daa0672d4805ba5cffa3d44b6d8f, State: Running, Role: LEADER
I20260928 20:23:49.930706 24647 consensus_queue.cc:237] T 00000000000000000000000000000000 P e849daa0672d4805ba5cffa3d44b6d8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e849daa0672d4805ba5cffa3d44b6d8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34323 } }
I20260928 20:23:49.931414 24649 sys_catalog.cc:455] T 00000000000000000000000000000000 P e849daa0672d4805ba5cffa3d44b6d8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e849daa0672d4805ba5cffa3d44b6d8f. Latest consensus state: current_term: 1 leader_uuid: "e849daa0672d4805ba5cffa3d44b6d8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e849daa0672d4805ba5cffa3d44b6d8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34323 } } }
I20260928 20:23:49.931519 24649 sys_catalog.cc:458] T 00000000000000000000000000000000 P e849daa0672d4805ba5cffa3d44b6d8f [sys.catalog]: This master's current role is: LEADER
I20260928 20:23:49.931578 24648 sys_catalog.cc:455] T 00000000000000000000000000000000 P e849daa0672d4805ba5cffa3d44b6d8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e849daa0672d4805ba5cffa3d44b6d8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e849daa0672d4805ba5cffa3d44b6d8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34323 } } }
I20260928 20:23:49.931649 24648 sys_catalog.cc:458] T 00000000000000000000000000000000 P e849daa0672d4805ba5cffa3d44b6d8f [sys.catalog]: This master's current role is: LEADER
I20260928 20:23:49.931816 24656 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:23:49.932449 24656 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:23:49.934620 24656 catalog_manager.cc:1347] Generated new cluster ID: 237feb7fc5a74bda9d53934c1bd38193
I20260928 20:23:49.934674 24656 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:23:49.947875 24656 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:23:49.948096 24656 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:23:49.952590 24656 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e849daa0672d4805ba5cffa3d44b6d8f: Generated new TSK 0
I20260928 20:23:49.952852 24656 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 20:23:49.962045 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 24569
2026-09-28T20:23:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1370 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790626764250411-3144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:23:50 dist-test-slave-b83v krb5kdc[24668](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:23:50 dist-test-slave-b83v krb5kdc[24668](info): set up 2 sockets
Sep 28 20:23:50 dist-test-slave-b83v krb5kdc[24668](info): commencing operation
krb5kdc: starting...
W20260928 20:23:51.012282 3144 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Sep 28 20:23:51 dist-test-slave-b83v krb5kdc[24668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:23:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:23:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.18.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab.
Entry for principal kudu/127.3.18.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab.
I20260928 20:23:51.139120 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790626764250411-3144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790626764250411-3144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790626764250411-3144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:35291
--webserver_interface=127.3.18.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:41855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.18.62:35291
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:23:51.224570 24681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:23:51.224845 24681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:23:51.224891 24681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:23:51.226290 24681 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:23:51.226377 24681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:23:51.226401 24681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:23:51.226423 24681 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:23:51.226455 24681 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:23:51.228235 24681 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:41855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790626764250411-3144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790626764250411-3144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.18.62:35291
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:35291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.18.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790626764250411-3144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:23:51.228585 24681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:23:51.228839 24681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:23:51.234481 24686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:51.234484 24687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:23:51.234756 24681 server_base.cc:1034] running on GCE node
W20260928 20:23:51.234484 24689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:23:51.235131 24681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:23:51.235484 24681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:23:51.236719 24681 hybrid_clock.cc:648] HybridClock initialized: now 1790627031236715 us; error 75 us; skew 500 ppm
Sep 28 20:23:51 dist-test-slave-b83v krb5kdc[24668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627031, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:23:51.239632 24681 init.cc:377] Logged in from keytab as kudu/127.3.18.62@KRBTEST.COM (short username kudu)
I20260928 20:23:51.240209 24681 webserver.cc:460] Webserver started at http://127.3.18.62:41411/ using document root <none> and password file <none>
I20260928 20:23:51.240437 24681 fs_manager.cc:359] Metadata directory not provided
I20260928 20:23:51.240480 24681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:23:51.240597 24681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:23:51.241698 24681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790626764250411-3144-0/minicluster-data/master-0/data/instance:
uuid: "5bc47c5ffc1e4b6bb40abfbc6a1afa7e"
format_stamp: "Formatted at 2026-09-28 20:23:51 on dist-test-slave-b83v"
server_key: "7af2e137de438bf99c4f615cdf826233"
server_key_iv: "3ccc01ea8e40b2593b89c4b041d204aa"
server_key_version: "encryptionkey@0"
I20260928 20:23:51.242027 24681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790626764250411-3144-0/minicluster-data/master-0/wal/instance:
uuid: "5bc47c5ffc1e4b6bb40abfbc6a1afa7e"
format_stamp: "Formatted at 2026-09-28 20:23:51 on dist-test-slave-b83v"
server_key: "7af2e137de438bf99c4f615cdf826233"
server_key_iv: "3ccc01ea8e40b2593b89c4b041d204aa"
server_key_version: "encryptionkey@0"
I20260928 20:23:51.243409 24681 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 20:23:51.244283 24696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:51.244515 24681 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 20:23:51.244694 24681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790626764250411-3144-0/minicluster-data/master-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790626764250411-3144-0/minicluster-data/master-0/wal
uuid: "5bc47c5ffc1e4b6bb40abfbc6a1afa7e"
format_stamp: "Formatted at 2026-09-28 20:23:51 on dist-test-slave-b83v"
server_key: "7af2e137de438bf99c4f615cdf826233"
server_key_iv: "3ccc01ea8e40b2593b89c4b041d204aa"
server_key_version: "encryptionkey@0"
I20260928 20:23:51.244773 24681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790626764250411-3144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790626764250411-3144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790626764250411-3144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:23:51.279848 24681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:23:51.282258 24681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:23:51.282405 24681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:23:51.286514 24681 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.62:35291
I20260928 20:23:51.286844 24748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.62:35291 every 8 connection(s)
I20260928 20:23:51.286911 24681 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
I20260928 20:23:51.287572 24749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:23:51.289909 24749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bc47c5ffc1e4b6bb40abfbc6a1afa7e: Bootstrap starting.
I20260928 20:23:51.290441 24749 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5bc47c5ffc1e4b6bb40abfbc6a1afa7e: Neither blocks nor log segments found. Creating new log.
I20260928 20:23:51.290674 24749 log.cc:824] T 00000000000000000000000000000000 P 5bc47c5ffc1e4b6bb40abfbc6a1afa7e: Log is configured to *not* fsync() on all Append() calls
I20260928 20:23:51.291359 24749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bc47c5ffc1e4b6bb40abfbc6a1afa7e: No bootstrap required, opened a new log
I20260928 20:23:51.292950 24749 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5bc47c5ffc1e4b6bb40abfbc6a1afa7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bc47c5ffc1e4b6bb40abfbc6a1afa7e" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 35291 } }
I20260928 20:23:51.293104 24749 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5bc47c5ffc1e4b6bb40abfbc6a1afa7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:23:51.293128 24749 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5bc47c5ffc1e4b6bb40abfbc6a1afa7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bc47c5ffc1e4b6bb40abfbc6a1afa7e, State: Initialized, Role: FOLLOWER
I20260928 20:23:51.293264 24749 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5bc47c5ffc1e4b6bb40abfbc6a1afa7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bc47c5ffc1e4b6bb40abfbc6a1afa7e" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 35291 } }
I20260928 20:23:51.293347 24749 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5bc47c5ffc1e4b6bb40abfbc6a1afa7e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:23:51.293370 24749 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5bc47c5ffc1e4b6bb40abfbc6a1afa7e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:23:51.293398 24749 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5bc47c5ffc1e4b6bb40abfbc6a1afa7e [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:23:51.294272 24749 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5bc47c5ffc1e4b6bb40abfbc6a1afa7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bc47c5ffc1e4b6bb40abfbc6a1afa7e" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 35291 } }
I20260928 20:23:51.294360 24749 leader_election.cc:302] T 00000000000000000000000000000000 P 5bc47c5ffc1e4b6bb40abfbc6a1afa7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5bc47c5ffc1e4b6bb40abfbc6a1afa7e; no voters:
I20260928 20:23:51.294523 24749 leader_election.cc:288] T 00000000000000000000000000000000 P 5bc47c5ffc1e4b6bb40abfbc6a1afa7e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:23:51.294549 24752 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5bc47c5ffc1e4b6bb40abfbc6a1afa7e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:23:51.294669 24752 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5bc47c5ffc1e4b6bb40abfbc6a1afa7e [term 1 LEADER]: Becoming Leader. State: Replica: 5bc47c5ffc1e4b6bb40abfbc6a1afa7e, State: Running, Role: LEADER
I20260928 20:23:51.294752 24752 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5bc47c5ffc1e4b6bb40abfbc6a1afa7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bc47c5ffc1e4b6bb40abfbc6a1afa7e" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 35291 } }
I20260928 20:23:51.294783 24749 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5bc47c5ffc1e4b6bb40abfbc6a1afa7e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:23:51.295238 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 24681
I20260928 20:23:51.295367 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790626764250411-3144-0/minicluster-data/master-0/wal/instance
I20260928 20:23:51.295568 3144 external_mini_cluster.cc:1428] Setting key 50d8cb1df469a1d3b6654b76f5a84819
I20260928 20:23:51.296221 24754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bc47c5ffc1e4b6bb40abfbc6a1afa7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bc47c5ffc1e4b6bb40abfbc6a1afa7e. Latest consensus state: current_term: 1 leader_uuid: "5bc47c5ffc1e4b6bb40abfbc6a1afa7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bc47c5ffc1e4b6bb40abfbc6a1afa7e" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 35291 } } }
I20260928 20:23:51.296339 24754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bc47c5ffc1e4b6bb40abfbc6a1afa7e [sys.catalog]: This master's current role is: LEADER
I20260928 20:23:51.296545 24753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bc47c5ffc1e4b6bb40abfbc6a1afa7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5bc47c5ffc1e4b6bb40abfbc6a1afa7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bc47c5ffc1e4b6bb40abfbc6a1afa7e" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 35291 } } }
I20260928 20:23:51.296707 24753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bc47c5ffc1e4b6bb40abfbc6a1afa7e [sys.catalog]: This master's current role is: LEADER
I20260928 20:23:51.296550 24771 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:23:51.297250 24771 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:23:51.298722 24771 catalog_manager.cc:1347] Generated new cluster ID: 68d6dc3cabed43a5907b515ed2ec754e
I20260928 20:23:51.298789 24771 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 20:23:51 dist-test-slave-b83v krb5kdc[24668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:23:51.309361 24768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:51.296147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34551 (local address 127.3.18.62:35291)
0928 20:23:51.296342 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:51.296354 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:51.296830 (+ 476us) server_negotiation.cc:407] Connection header received
0928 20:23:51.296954 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:51.296962 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:51.297039 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:51.297181 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:51.298089 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:51.299476 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:51.300629 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:51.301267 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:51.301371 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:51.304702 (+ 3331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:51.304707 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:51.304707 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:23:51.304722 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:51.306896 (+ 2174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:51.307425 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:51.307429 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:51.307429 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:51.307565 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:51.307874 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:51.307875 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:51.307875 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:51.308258 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:51.308432 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:51.308993 (+ 561us) server_negotiation.cc:299] Negotiation successful
0928 20:23:51.309106 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":101,"threads_started":1}
I20260928 20:23:51.310580 3144 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 20:23:51.320021 24768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:51.312216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52355 (local address 127.3.18.62:35291)
0928 20:23:51.312252 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:51.312254 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:51.312263 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:23:51.312314 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:51.312315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:51.312324 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:51.312384 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:51.313074 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:51.313883 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:51.314865 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:51.315258 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:51.315339 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:51.316883 (+ 1544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:51.316885 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:51.316885 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:23:51.316892 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:51.318580 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:51.319007 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:51.319008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:51.319008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:51.319072 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:51.319297 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:51.319298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:51.319298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:51.319438 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:51.319550 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:51.319847 (+ 297us) server_negotiation.cc:299] Negotiation successful
0928 20:23:51.319918 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 20:23:51.337083 24771 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:23:51.337648 24771 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:23:51.340922 24771 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5bc47c5ffc1e4b6bb40abfbc6a1afa7e: Generated new TSK 0
I20260928 20:23:51.341197 24771 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 20:23:51.349812 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 24681
W20260928 20:23:51.354358 24666 connection.cc:535] client connection to 127.3.18.62:35291 recv error: Network error: failed to read from TLS socket (remote: 127.3.18.62:35291): Connection reset by peer (error 104)
2026-09-28T20:23:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1387 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790626764250411-3144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:23:51 dist-test-slave-b83v krb5kdc[24781](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:23:51 dist-test-slave-b83v krb5kdc[24781](info): set up 2 sockets
Sep 28 20:23:51 dist-test-slave-b83v krb5kdc[24781](info): commencing operation
krb5kdc: starting...
W20260928 20:23:52.428318 3144 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Sep 28 20:23:52 dist-test-slave-b83v krb5kdc[24781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:23:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:23:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.18.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab.
Entry for principal kudu/127.3.18.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab.
I20260928 20:23:52.559480 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790626764250411-3144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790626764250411-3144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790626764250411-3144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:46793
--webserver_interface=127.3.18.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:34407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.18.62:46793
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:23:52.642297 24794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:23:52.642592 24794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:23:52.642648 24794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:23:52.643980 24794 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:23:52.644049 24794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:23:52.644073 24794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:23:52.644094 24794 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:23:52.644126 24794 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:23:52.646250 24794 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:34407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790626764250411-3144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790626764250411-3144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.18.62:46793
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:46793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.18.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790626764250411-3144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:23:52.646577 24794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:23:52.646818 24794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:23:52.652148 24800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:52.652158 24802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:52.652189 24799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:23:52.652555 24794 server_base.cc:1034] running on GCE node
I20260928 20:23:52.652845 24794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:23:52.653148 24794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:23:52.654330 24794 hybrid_clock.cc:648] HybridClock initialized: now 1790627032654315 us; error 35 us; skew 500 ppm
Sep 28 20:23:52 dist-test-slave-b83v krb5kdc[24781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627032, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:23:52.657264 24794 init.cc:377] Logged in from keytab as kudu/127.3.18.62@KRBTEST.COM (short username kudu)
I20260928 20:23:52.657936 24794 webserver.cc:460] Webserver started at http://127.3.18.62:33057/ using document root <none> and password file <none>
I20260928 20:23:52.658209 24794 fs_manager.cc:359] Metadata directory not provided
I20260928 20:23:52.658290 24794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:23:52.658461 24794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:23:52.659685 24794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790626764250411-3144-0/minicluster-data/master-0/data/instance:
uuid: "3e102f87ab5a46c2bdc930cebe7f2689"
format_stamp: "Formatted at 2026-09-28 20:23:52 on dist-test-slave-b83v"
server_key: "ed52b375f0dad83781c0815e818e9d13"
server_key_iv: "9acaf64bbe4942ccbdee1f6ac9067787"
server_key_version: "encryptionkey@0"
I20260928 20:23:52.660194 24794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790626764250411-3144-0/minicluster-data/master-0/wal/instance:
uuid: "3e102f87ab5a46c2bdc930cebe7f2689"
format_stamp: "Formatted at 2026-09-28 20:23:52 on dist-test-slave-b83v"
server_key: "ed52b375f0dad83781c0815e818e9d13"
server_key_iv: "9acaf64bbe4942ccbdee1f6ac9067787"
server_key_version: "encryptionkey@0"
I20260928 20:23:52.661886 24794 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 20:23:52.663036 24809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:52.663321 24794 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:23:52.663451 24794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790626764250411-3144-0/minicluster-data/master-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790626764250411-3144-0/minicluster-data/master-0/wal
uuid: "3e102f87ab5a46c2bdc930cebe7f2689"
format_stamp: "Formatted at 2026-09-28 20:23:52 on dist-test-slave-b83v"
server_key: "ed52b375f0dad83781c0815e818e9d13"
server_key_iv: "9acaf64bbe4942ccbdee1f6ac9067787"
server_key_version: "encryptionkey@0"
I20260928 20:23:52.663614 24794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790626764250411-3144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790626764250411-3144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790626764250411-3144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:23:52.677169 24794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:23:52.679593 24794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:23:52.679895 24794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:23:52.684221 24794 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.62:46793
I20260928 20:23:52.684254 24861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.62:46793 every 8 connection(s)
I20260928 20:23:52.684633 24794 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
I20260928 20:23:52.685170 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 24794
I20260928 20:23:52.685334 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790626764250411-3144-0/minicluster-data/master-0/wal/instance
I20260928 20:23:52.685631 3144 external_mini_cluster.cc:1428] Setting key c778995fdaf0f21dabeaab74aba4b739
I20260928 20:23:52.685854 24862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:23:52.689471 24862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e102f87ab5a46c2bdc930cebe7f2689: Bootstrap starting.
I20260928 20:23:52.690140 24862 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e102f87ab5a46c2bdc930cebe7f2689: Neither blocks nor log segments found. Creating new log.
I20260928 20:23:52.690428 24862 log.cc:824] T 00000000000000000000000000000000 P 3e102f87ab5a46c2bdc930cebe7f2689: Log is configured to *not* fsync() on all Append() calls
I20260928 20:23:52.691126 24862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e102f87ab5a46c2bdc930cebe7f2689: No bootstrap required, opened a new log
I20260928 20:23:52.692718 24865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:52.686501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34397 (local address 127.3.18.62:46793)
0928 20:23:52.686712 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:52.686722 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:52.686757 (+ 35us) server_negotiation.cc:407] Connection header received
0928 20:23:52.687471 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:52.687481 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:52.687557 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:52.687724 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:52.688046 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:52.689411 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:52.690333 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:52.690851 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:52.690980 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:52.691842 (+ 862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:52.691847 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:52.691847 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:23:52.692076 (+ 229us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:52.692111 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:23:52.692123 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:52.692233 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:52.692352 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 20:23:52.692384 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":89,"threads_started":1}
I20260928 20:23:52.692876 24862 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e102f87ab5a46c2bdc930cebe7f2689 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e102f87ab5a46c2bdc930cebe7f2689" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 46793 } }
I20260928 20:23:52.693068 24862 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e102f87ab5a46c2bdc930cebe7f2689 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:23:52.693109 24862 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e102f87ab5a46c2bdc930cebe7f2689 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e102f87ab5a46c2bdc930cebe7f2689, State: Initialized, Role: FOLLOWER
I20260928 20:23:52.693225 24862 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e102f87ab5a46c2bdc930cebe7f2689 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e102f87ab5a46c2bdc930cebe7f2689" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 46793 } }
I20260928 20:23:52.693326 24862 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e102f87ab5a46c2bdc930cebe7f2689 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:23:52.693362 24862 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e102f87ab5a46c2bdc930cebe7f2689 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:23:52.693413 24862 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e102f87ab5a46c2bdc930cebe7f2689 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:23:52.693805 3144 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 20:23:52.694164 24862 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e102f87ab5a46c2bdc930cebe7f2689 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e102f87ab5a46c2bdc930cebe7f2689" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 46793 } }
I20260928 20:23:52.694279 24862 leader_election.cc:302] T 00000000000000000000000000000000 P 3e102f87ab5a46c2bdc930cebe7f2689 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e102f87ab5a46c2bdc930cebe7f2689; no voters:
I20260928 20:23:52.694451 24862 leader_election.cc:288] T 00000000000000000000000000000000 P 3e102f87ab5a46c2bdc930cebe7f2689 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:23:52.694546 24867 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e102f87ab5a46c2bdc930cebe7f2689 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:23:52.694711 24862 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e102f87ab5a46c2bdc930cebe7f2689 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:23:52.694754 24867 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e102f87ab5a46c2bdc930cebe7f2689 [term 1 LEADER]: Becoming Leader. State: Replica: 3e102f87ab5a46c2bdc930cebe7f2689, State: Running, Role: LEADER
I20260928 20:23:52.694939 24867 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e102f87ab5a46c2bdc930cebe7f2689 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e102f87ab5a46c2bdc930cebe7f2689" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 46793 } }
I20260928 20:23:52.695492 24869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e102f87ab5a46c2bdc930cebe7f2689 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e102f87ab5a46c2bdc930cebe7f2689. Latest consensus state: current_term: 1 leader_uuid: "3e102f87ab5a46c2bdc930cebe7f2689" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e102f87ab5a46c2bdc930cebe7f2689" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 46793 } } }
I20260928 20:23:52.695720 24869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e102f87ab5a46c2bdc930cebe7f2689 [sys.catalog]: This master's current role is: LEADER
I20260928 20:23:52.695462 24868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e102f87ab5a46c2bdc930cebe7f2689 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e102f87ab5a46c2bdc930cebe7f2689" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e102f87ab5a46c2bdc930cebe7f2689" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 46793 } } }
I20260928 20:23:52.696015 24868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e102f87ab5a46c2bdc930cebe7f2689 [sys.catalog]: This master's current role is: LEADER
I20260928 20:23:52.696038 24880 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:23:52.696561 24880 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:23:52.699425 24865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:52.695717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34179 (local address 127.3.18.62:46793)
0928 20:23:52.695755 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:52.695757 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:52.696068 (+ 311us) server_negotiation.cc:407] Connection header received
0928 20:23:52.696182 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:52.696183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:52.696194 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:52.696227 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:52.696480 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:52.697154 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:52.697940 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:52.698261 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:52.698339 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:52.699121 (+ 782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:52.699122 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:52.699122 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:23:52.699127 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:52.699144 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:23:52.699150 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:52.699224 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:52.699331 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 20:23:52.699346 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 20:23:52.699432 24880 catalog_manager.cc:1347] Generated new cluster ID: 6bd30541efe145c689546b9872f3d558
I20260928 20:23:52.699517 24880 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:23:52.705741 24880 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:23:52.706305 24880 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:23:52.715034 24880 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e102f87ab5a46c2bdc930cebe7f2689: Generated new TSK 0
I20260928 20:23:52.715301 24880 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 20:23:52.723944 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 24794
W20260928 20:23:52.728744 24779 connection.cc:535] client connection to 127.3.18.62:46793 recv error: Network error: failed to read from TLS socket (remote: 127.3.18.62:46793): Connection reset by peer (error 104)
2026-09-28T20:23:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1374 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790626764250411-3144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:23:52 dist-test-slave-b83v krb5kdc[24893](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:23:52 dist-test-slave-b83v krb5kdc[24893](info): set up 2 sockets
Sep 28 20:23:52 dist-test-slave-b83v krb5kdc[24893](info): commencing operation
krb5kdc: starting...
W20260928 20:23:53.788426 3144 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Sep 28 20:23:53 dist-test-slave-b83v krb5kdc[24893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:23:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:23:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.18.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab.
Entry for principal kudu/127.3.18.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab.
I20260928 20:23:53.920097 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790626764250411-3144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790626764250411-3144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790626764250411-3144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:42945
--webserver_interface=127.3.18.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:38073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.18.62:42945
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:23:54.005105 24906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:23:54.005308 24906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:23:54.005332 24906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:23:54.006405 24906 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:23:54.006444 24906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:23:54.006459 24906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:23:54.006472 24906 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:23:54.006484 24906 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:23:54.007932 24906 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:38073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790626764250411-3144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790626764250411-3144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.18.62:42945
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:42945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.18.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790626764250411-3144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:23:54.008163 24906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:23:54.008359 24906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:23:54.013794 24911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:54.013800 24912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:54.013798 24914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:23:54.013847 24906 server_base.cc:1034] running on GCE node
I20260928 20:23:54.014303 24906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:23:54.014649 24906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:23:54.015882 24906 hybrid_clock.cc:648] HybridClock initialized: now 1790627034015834 us; error 54 us; skew 500 ppm
Sep 28 20:23:54 dist-test-slave-b83v krb5kdc[24893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627034, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:23:54.024744 24906 init.cc:377] Logged in from keytab as kudu/127.3.18.62@KRBTEST.COM (short username kudu)
I20260928 20:23:54.025385 24906 webserver.cc:460] Webserver started at http://127.3.18.62:35981/ using document root <none> and password file <none>
I20260928 20:23:54.025669 24906 fs_manager.cc:359] Metadata directory not provided
I20260928 20:23:54.025746 24906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:23:54.025905 24906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:23:54.027055 24906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790626764250411-3144-0/minicluster-data/master-0/data/instance:
uuid: "5a93c60837b34b89924b44c8ad3e435b"
format_stamp: "Formatted at 2026-09-28 20:23:54 on dist-test-slave-b83v"
server_key: "8e088db1a6ea6215fb28b5b5ad109dbe"
server_key_iv: "2a6531e5f8d5ef1d5836fdd763339437"
server_key_version: "encryptionkey@0"
I20260928 20:23:54.027509 24906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790626764250411-3144-0/minicluster-data/master-0/wal/instance:
uuid: "5a93c60837b34b89924b44c8ad3e435b"
format_stamp: "Formatted at 2026-09-28 20:23:54 on dist-test-slave-b83v"
server_key: "8e088db1a6ea6215fb28b5b5ad109dbe"
server_key_iv: "2a6531e5f8d5ef1d5836fdd763339437"
server_key_version: "encryptionkey@0"
I20260928 20:23:54.029116 24906 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 20:23:54.030004 24921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:54.030356 24906 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:23:54.030478 24906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790626764250411-3144-0/minicluster-data/master-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790626764250411-3144-0/minicluster-data/master-0/wal
uuid: "5a93c60837b34b89924b44c8ad3e435b"
format_stamp: "Formatted at 2026-09-28 20:23:54 on dist-test-slave-b83v"
server_key: "8e088db1a6ea6215fb28b5b5ad109dbe"
server_key_iv: "2a6531e5f8d5ef1d5836fdd763339437"
server_key_version: "encryptionkey@0"
I20260928 20:23:54.030599 24906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790626764250411-3144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790626764250411-3144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790626764250411-3144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:23:54.034391 24906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:23:54.036731 24906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:23:54.036902 24906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:23:54.041565 24906 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.62:42945
I20260928 20:23:54.041606 24973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.62:42945 every 8 connection(s)
I20260928 20:23:54.042002 24906 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
I20260928 20:23:54.043217 24974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:23:54.045454 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 24906
I20260928 20:23:54.045603 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790626764250411-3144-0/minicluster-data/master-0/wal/instance
I20260928 20:23:54.045792 3144 external_mini_cluster.cc:1428] Setting key a422a79b8cc0483fd1029f9f873ab794
I20260928 20:23:54.046030 24974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a93c60837b34b89924b44c8ad3e435b: Bootstrap starting.
I20260928 20:23:54.046823 24974 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a93c60837b34b89924b44c8ad3e435b: Neither blocks nor log segments found. Creating new log.
I20260928 20:23:54.047176 24974 log.cc:824] T 00000000000000000000000000000000 P 5a93c60837b34b89924b44c8ad3e435b: Log is configured to *not* fsync() on all Append() calls
I20260928 20:23:54.048077 24974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a93c60837b34b89924b44c8ad3e435b: No bootstrap required, opened a new log
I20260928 20:23:54.048897 24977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:54.046378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51739 (local address 127.3.18.62:42945)
0928 20:23:54.046610 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:54.046621 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:54.046659 (+ 38us) server_negotiation.cc:407] Connection header received
0928 20:23:54.047291 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:54.047301 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:54.047340 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:54.047506 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:54.048386 (+ 880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:54.048390 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:54.048391 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:23:54.048395 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:54.048445 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:23:54.048459 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:54.048563 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:54.048638 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 20:23:54.048684 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":103,"threads_started":1}
I20260928 20:23:54.050204 3144 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 20:23:54.050590 24974 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a93c60837b34b89924b44c8ad3e435b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a93c60837b34b89924b44c8ad3e435b" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 42945 } }
I20260928 20:23:54.050776 24974 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a93c60837b34b89924b44c8ad3e435b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:23:54.050822 24974 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a93c60837b34b89924b44c8ad3e435b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a93c60837b34b89924b44c8ad3e435b, State: Initialized, Role: FOLLOWER
I20260928 20:23:54.050953 24974 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a93c60837b34b89924b44c8ad3e435b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a93c60837b34b89924b44c8ad3e435b" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 42945 } }
I20260928 20:23:54.051049 24974 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a93c60837b34b89924b44c8ad3e435b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:23:54.051095 24974 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a93c60837b34b89924b44c8ad3e435b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:23:54.051146 24974 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a93c60837b34b89924b44c8ad3e435b [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:23:54.051960 24974 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a93c60837b34b89924b44c8ad3e435b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a93c60837b34b89924b44c8ad3e435b" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 42945 } }
I20260928 20:23:54.052098 24974 leader_election.cc:302] T 00000000000000000000000000000000 P 5a93c60837b34b89924b44c8ad3e435b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a93c60837b34b89924b44c8ad3e435b; no voters:
I20260928 20:23:54.052282 24974 leader_election.cc:288] T 00000000000000000000000000000000 P 5a93c60837b34b89924b44c8ad3e435b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:23:54.052307 24984 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a93c60837b34b89924b44c8ad3e435b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:23:54.052491 24984 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a93c60837b34b89924b44c8ad3e435b [term 1 LEADER]: Becoming Leader. State: Replica: 5a93c60837b34b89924b44c8ad3e435b, State: Running, Role: LEADER
I20260928 20:23:54.052627 24974 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a93c60837b34b89924b44c8ad3e435b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:23:54.052628 24984 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a93c60837b34b89924b44c8ad3e435b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a93c60837b34b89924b44c8ad3e435b" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 42945 } }
I20260928 20:23:54.053192 24977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:54.051789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53077 (local address 127.3.18.62:42945)
0928 20:23:54.051832 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:54.051834 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:54.051962 (+ 128us) server_negotiation.cc:407] Connection header received
0928 20:23:54.052007 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:54.052008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:54.052015 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:54.052088 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:54.052793 (+ 705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:54.052794 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:54.052795 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:23:54.052799 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:54.052815 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:23:54.052819 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:54.052884 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:54.053086 (+ 202us) server_negotiation.cc:299] Negotiation successful
0928 20:23:54.053100 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 20:23:54.054049 24986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a93c60837b34b89924b44c8ad3e435b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a93c60837b34b89924b44c8ad3e435b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a93c60837b34b89924b44c8ad3e435b" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 42945 } } }
I20260928 20:23:54.054136 24986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a93c60837b34b89924b44c8ad3e435b [sys.catalog]: This master's current role is: LEADER
I20260928 20:23:54.054409 24985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a93c60837b34b89924b44c8ad3e435b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a93c60837b34b89924b44c8ad3e435b. Latest consensus state: current_term: 1 leader_uuid: "5a93c60837b34b89924b44c8ad3e435b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a93c60837b34b89924b44c8ad3e435b" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 42945 } } }
I20260928 20:23:54.054589 24985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a93c60837b34b89924b44c8ad3e435b [sys.catalog]: This master's current role is: LEADER
W20260928 20:23:54.054858 24999 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5a93c60837b34b89924b44c8ad3e435b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 20:23:54.054942 24999 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 20:23:54.055007 24998 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:23:54.055218 24998 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:23:54.057009 24998 catalog_manager.cc:1347] Generated new cluster ID: 7acc7eb79fe941fc8abf32cc147723a5
I20260928 20:23:54.057087 24998 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:23:54.066558 24998 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:23:54.066813 24998 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:23:54.071293 24998 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a93c60837b34b89924b44c8ad3e435b: Generated new TSK 0
I20260928 20:23:54.071552 24998 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 20:23:54.077354 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 24906
2026-09-28T20:23:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1352 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790626764250411-3144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:23:54 dist-test-slave-b83v krb5kdc[25005](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:23:54 dist-test-slave-b83v krb5kdc[25005](info): set up 2 sockets
Sep 28 20:23:54 dist-test-slave-b83v krb5kdc[25005](info): commencing operation
krb5kdc: starting...
W20260928 20:23:55.144364 3144 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Sep 28 20:23:55 dist-test-slave-b83v krb5kdc[25005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:23:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:23:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.18.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab.
Entry for principal kudu/127.3.18.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab.
I20260928 20:23:55.279873 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790626764250411-3144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790626764250411-3144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790626764250411-3144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:35089
--webserver_interface=127.3.18.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:33195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.18.62:35089
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:23:55.363176 25018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:23:55.363459 25018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:23:55.363499 25018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:23:55.364751 25018 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:23:55.364826 25018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:23:55.364851 25018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:23:55.364885 25018 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:23:55.364912 25018 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:23:55.366461 25018 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:33195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790626764250411-3144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790626764250411-3144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.18.62:35089
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:35089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.18.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790626764250411-3144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:23:55.366768 25018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:23:55.367012 25018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:23:55.372329 25024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:55.372329 25026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:23:55.372454 25018 server_base.cc:1034] running on GCE node
W20260928 20:23:55.372332 25023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:23:55.372879 25018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:23:55.373188 25018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:23:55.374392 25018 hybrid_clock.cc:648] HybridClock initialized: now 1790627035374387 us; error 41 us; skew 500 ppm
Sep 28 20:23:55 dist-test-slave-b83v krb5kdc[25005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627035, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:23:55.377262 25018 init.cc:377] Logged in from keytab as kudu/127.3.18.62@KRBTEST.COM (short username kudu)
I20260928 20:23:55.377735 25018 webserver.cc:460] Webserver started at http://127.3.18.62:41309/ using document root <none> and password file <none>
I20260928 20:23:55.377985 25018 fs_manager.cc:359] Metadata directory not provided
I20260928 20:23:55.378031 25018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:23:55.378201 25018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:23:55.379309 25018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790626764250411-3144-0/minicluster-data/master-0/data/instance:
uuid: "af66c347ea474d63bdc3d8fd597a973d"
format_stamp: "Formatted at 2026-09-28 20:23:55 on dist-test-slave-b83v"
server_key: "61aa626bd72fc9a07a8bf545c7a2d1b2"
server_key_iv: "37908567a8cc4ecba81c2914abf27393"
server_key_version: "encryptionkey@0"
I20260928 20:23:55.379683 25018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790626764250411-3144-0/minicluster-data/master-0/wal/instance:
uuid: "af66c347ea474d63bdc3d8fd597a973d"
format_stamp: "Formatted at 2026-09-28 20:23:55 on dist-test-slave-b83v"
server_key: "61aa626bd72fc9a07a8bf545c7a2d1b2"
server_key_iv: "37908567a8cc4ecba81c2914abf27393"
server_key_version: "encryptionkey@0"
I20260928 20:23:55.381162 25018 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:23:55.382220 25033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:55.382630 25018 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:23:55.382720 25018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790626764250411-3144-0/minicluster-data/master-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790626764250411-3144-0/minicluster-data/master-0/wal
uuid: "af66c347ea474d63bdc3d8fd597a973d"
format_stamp: "Formatted at 2026-09-28 20:23:55 on dist-test-slave-b83v"
server_key: "61aa626bd72fc9a07a8bf545c7a2d1b2"
server_key_iv: "37908567a8cc4ecba81c2914abf27393"
server_key_version: "encryptionkey@0"
I20260928 20:23:55.382822 25018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790626764250411-3144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790626764250411-3144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790626764250411-3144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:23:55.395409 25018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:23:55.397529 25018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:23:55.397678 25018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:23:55.401834 25018 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.62:35089
I20260928 20:23:55.401892 25085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.62:35089 every 8 connection(s)
I20260928 20:23:55.402264 25018 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
I20260928 20:23:55.403151 25086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:23:55.405289 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 25018
I20260928 20:23:55.405416 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790626764250411-3144-0/minicluster-data/master-0/wal/instance
I20260928 20:23:55.405628 3144 external_mini_cluster.cc:1428] Setting key 4b804841fd05e38a50a1df6fed88fb98
I20260928 20:23:55.406126 25086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af66c347ea474d63bdc3d8fd597a973d: Bootstrap starting.
I20260928 20:23:55.406877 25086 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af66c347ea474d63bdc3d8fd597a973d: Neither blocks nor log segments found. Creating new log.
I20260928 20:23:55.407315 25086 log.cc:824] T 00000000000000000000000000000000 P af66c347ea474d63bdc3d8fd597a973d: Log is configured to *not* fsync() on all Append() calls
I20260928 20:23:55.408134 25086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af66c347ea474d63bdc3d8fd597a973d: No bootstrap required, opened a new log
I20260928 20:23:55.410367 25086 raft_consensus.cc:348] T 00000000000000000000000000000000 P af66c347ea474d63bdc3d8fd597a973d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af66c347ea474d63bdc3d8fd597a973d" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 35089 } }
I20260928 20:23:55.410534 25086 raft_consensus.cc:374] T 00000000000000000000000000000000 P af66c347ea474d63bdc3d8fd597a973d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:23:55.410622 25086 raft_consensus.cc:729] T 00000000000000000000000000000000 P af66c347ea474d63bdc3d8fd597a973d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af66c347ea474d63bdc3d8fd597a973d, State: Initialized, Role: FOLLOWER
I20260928 20:23:55.410846 25086 consensus_queue.cc:260] T 00000000000000000000000000000000 P af66c347ea474d63bdc3d8fd597a973d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af66c347ea474d63bdc3d8fd597a973d" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 35089 } }
I20260928 20:23:55.410953 25086 raft_consensus.cc:388] T 00000000000000000000000000000000 P af66c347ea474d63bdc3d8fd597a973d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:23:55.410993 25086 raft_consensus.cc:482] T 00000000000000000000000000000000 P af66c347ea474d63bdc3d8fd597a973d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:23:55.411051 25086 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af66c347ea474d63bdc3d8fd597a973d [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:23:55.411957 25086 raft_consensus.cc:504] T 00000000000000000000000000000000 P af66c347ea474d63bdc3d8fd597a973d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af66c347ea474d63bdc3d8fd597a973d" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 35089 } }
I20260928 20:23:55.412113 25086 leader_election.cc:302] T 00000000000000000000000000000000 P af66c347ea474d63bdc3d8fd597a973d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af66c347ea474d63bdc3d8fd597a973d; no voters:
I20260928 20:23:55.412286 25086 leader_election.cc:288] T 00000000000000000000000000000000 P af66c347ea474d63bdc3d8fd597a973d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:23:55.412343 25091 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af66c347ea474d63bdc3d8fd597a973d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:23:55.412556 25091 raft_consensus.cc:686] T 00000000000000000000000000000000 P af66c347ea474d63bdc3d8fd597a973d [term 1 LEADER]: Becoming Leader. State: Replica: af66c347ea474d63bdc3d8fd597a973d, State: Running, Role: LEADER
I20260928 20:23:55.412570 25086 sys_catalog.cc:564] T 00000000000000000000000000000000 P af66c347ea474d63bdc3d8fd597a973d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:23:55.412777 25091 consensus_queue.cc:237] T 00000000000000000000000000000000 P af66c347ea474d63bdc3d8fd597a973d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af66c347ea474d63bdc3d8fd597a973d" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 35089 } }
Sep 28 20:23:55 dist-test-slave-b83v krb5kdc[25005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:23:55.413314 25093 sys_catalog.cc:455] T 00000000000000000000000000000000 P af66c347ea474d63bdc3d8fd597a973d [sys.catalog]: SysCatalogTable state changed. Reason: New leader af66c347ea474d63bdc3d8fd597a973d. Latest consensus state: current_term: 1 leader_uuid: "af66c347ea474d63bdc3d8fd597a973d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af66c347ea474d63bdc3d8fd597a973d" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 35089 } } }
I20260928 20:23:55.413419 25093 sys_catalog.cc:458] T 00000000000000000000000000000000 P af66c347ea474d63bdc3d8fd597a973d [sys.catalog]: This master's current role is: LEADER
I20260928 20:23:55.413471 25092 sys_catalog.cc:455] T 00000000000000000000000000000000 P af66c347ea474d63bdc3d8fd597a973d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af66c347ea474d63bdc3d8fd597a973d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af66c347ea474d63bdc3d8fd597a973d" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 35089 } } }
I20260928 20:23:55.413537 25092 sys_catalog.cc:458] T 00000000000000000000000000000000 P af66c347ea474d63bdc3d8fd597a973d [sys.catalog]: This master's current role is: LEADER
I20260928 20:23:55.413849 25096 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:23:55.414386 25096 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:23:55.417068 25096 catalog_manager.cc:1347] Generated new cluster ID: 7e41b09a546e48b997097ab43281e1dc
I20260928 20:23:55.417125 25096 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:23:55.418081 25089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:55.406190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40299 (local address 127.3.18.62:35089)
0928 20:23:55.406402 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:55.406411 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:55.406437 (+ 26us) server_negotiation.cc:407] Connection header received
0928 20:23:55.406893 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:55.406901 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:55.406951 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:55.407126 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:55.407971 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:55.408969 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:55.410132 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:55.410545 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:55.410761 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:55.413700 (+ 2939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:55.413705 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:55.413706 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:23:55.413720 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:55.416167 (+ 2447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:55.416670 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:55.416674 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:55.416674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:55.416788 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:55.417039 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:55.417040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:55.417041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:55.417375 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:55.417527 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:55.417681 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 20:23:55.417834 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":125,"threads_started":1}
I20260928 20:23:55.419027 3144 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 20:23:55.425747 25096 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:23:55.426434 25096 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:23:55.428791 25089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:55.420557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43241 (local address 127.3.18.62:35089)
0928 20:23:55.420597 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:55.420599 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:55.420713 (+ 114us) server_negotiation.cc:407] Connection header received
0928 20:23:55.420806 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:55.420806 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:55.420816 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:55.420847 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:55.421648 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:55.422450 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:55.423521 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:55.423875 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:55.423921 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:55.425903 (+ 1982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:55.425904 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:55.425904 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:23:55.425911 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:55.427547 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:55.427978 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:55.427979 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:55.427979 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:55.428038 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:55.428263 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:55.428263 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:55.428264 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:55.428378 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:55.428464 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:55.428604 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 20:23:55.428690 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 20:23:55.435691 25096 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af66c347ea474d63bdc3d8fd597a973d: Generated new TSK 0
I20260928 20:23:55.435925 25096 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 20:23:55.440501 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 25018
2026-09-28T20:23:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1362 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790626764250411-3144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:23:55 dist-test-slave-b83v krb5kdc[25117](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:23:55 dist-test-slave-b83v krb5kdc[25117](info): set up 2 sockets
Sep 28 20:23:55 dist-test-slave-b83v krb5kdc[25117](info): commencing operation
krb5kdc: starting...
W20260928 20:23:56.476488 3144 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Sep 28 20:23:56 dist-test-slave-b83v krb5kdc[25117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:23:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:23:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.18.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab.
Entry for principal kudu/127.3.18.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab.
I20260928 20:23:56.599694 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790626764250411-3144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790626764250411-3144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790626764250411-3144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:41253
--webserver_interface=127.3.18.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:34987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.18.62:41253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:23:56.684533 25130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:23:56.684780 25130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:23:56.684805 25130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:23:56.685945 25130 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:23:56.685992 25130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:23:56.686007 25130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:23:56.686019 25130 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:23:56.686031 25130 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:23:56.687400 25130 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:34987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790626764250411-3144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790626764250411-3144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.18.62:41253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:41253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.18.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790626764250411-3144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:23:56.687593 25130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:23:56.687819 25130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 20:23:56.693039 25130 server_base.cc:1034] running on GCE node
W20260928 20:23:56.692909 25138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:56.692909 25135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:56.693007 25136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:23:56.693393 25130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:23:56.693753 25130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:23:56.694919 25130 hybrid_clock.cc:648] HybridClock initialized: now 1790627036694908 us; error 30 us; skew 500 ppm
Sep 28 20:23:56 dist-test-slave-b83v krb5kdc[25117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627036, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:23:56.697660 25130 init.cc:377] Logged in from keytab as kudu/127.3.18.62@KRBTEST.COM (short username kudu)
I20260928 20:23:56.698191 25130 webserver.cc:460] Webserver started at http://127.3.18.62:44125/ using document root <none> and password file <none>
I20260928 20:23:56.698429 25130 fs_manager.cc:359] Metadata directory not provided
I20260928 20:23:56.698498 25130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:23:56.698639 25130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:23:56.699739 25130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790626764250411-3144-0/minicluster-data/master-0/data/instance:
uuid: "8c211bb17b984b04b28cea328cb09c9b"
format_stamp: "Formatted at 2026-09-28 20:23:56 on dist-test-slave-b83v"
server_key: "bfa35b279029f08f83e13c4b6120a5a0"
server_key_iv: "617cfd9e4fffc807cf3ef8470fa755b9"
server_key_version: "encryptionkey@0"
I20260928 20:23:56.700197 25130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790626764250411-3144-0/minicluster-data/master-0/wal/instance:
uuid: "8c211bb17b984b04b28cea328cb09c9b"
format_stamp: "Formatted at 2026-09-28 20:23:56 on dist-test-slave-b83v"
server_key: "bfa35b279029f08f83e13c4b6120a5a0"
server_key_iv: "617cfd9e4fffc807cf3ef8470fa755b9"
server_key_version: "encryptionkey@0"
I20260928 20:23:56.701581 25130 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:23:56.702410 25145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:56.702594 25130 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:23:56.702695 25130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790626764250411-3144-0/minicluster-data/master-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790626764250411-3144-0/minicluster-data/master-0/wal
uuid: "8c211bb17b984b04b28cea328cb09c9b"
format_stamp: "Formatted at 2026-09-28 20:23:56 on dist-test-slave-b83v"
server_key: "bfa35b279029f08f83e13c4b6120a5a0"
server_key_iv: "617cfd9e4fffc807cf3ef8470fa755b9"
server_key_version: "encryptionkey@0"
I20260928 20:23:56.702777 25130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790626764250411-3144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790626764250411-3144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790626764250411-3144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:23:56.718793 25130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:23:56.721405 25130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:23:56.721676 25130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:23:56.726987 25130 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.62:41253
I20260928 20:23:56.727025 25197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.62:41253 every 8 connection(s)
I20260928 20:23:56.727612 25130 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
I20260928 20:23:56.728477 25198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:23:56.730921 25198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c211bb17b984b04b28cea328cb09c9b: Bootstrap starting.
I20260928 20:23:56.731503 25198 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c211bb17b984b04b28cea328cb09c9b: Neither blocks nor log segments found. Creating new log.
I20260928 20:23:56.731765 25198 log.cc:824] T 00000000000000000000000000000000 P 8c211bb17b984b04b28cea328cb09c9b: Log is configured to *not* fsync() on all Append() calls
I20260928 20:23:56.732630 25198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c211bb17b984b04b28cea328cb09c9b: No bootstrap required, opened a new log
I20260928 20:23:56.734292 25198 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c211bb17b984b04b28cea328cb09c9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c211bb17b984b04b28cea328cb09c9b" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 41253 } }
I20260928 20:23:56.734449 25198 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c211bb17b984b04b28cea328cb09c9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:23:56.734472 25198 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c211bb17b984b04b28cea328cb09c9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c211bb17b984b04b28cea328cb09c9b, State: Initialized, Role: FOLLOWER
I20260928 20:23:56.734587 25198 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c211bb17b984b04b28cea328cb09c9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c211bb17b984b04b28cea328cb09c9b" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 41253 } }
I20260928 20:23:56.734687 25198 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c211bb17b984b04b28cea328cb09c9b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:23:56.734727 25198 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c211bb17b984b04b28cea328cb09c9b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:23:56.734783 25198 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c211bb17b984b04b28cea328cb09c9b [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:23:56.735239 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 25130
I20260928 20:23:56.735366 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790626764250411-3144-0/minicluster-data/master-0/wal/instance
I20260928 20:23:56.735570 3144 external_mini_cluster.cc:1428] Setting key 9589710dba03daa5a9cb16614b0a8f8a
I20260928 20:23:56.736269 25198 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c211bb17b984b04b28cea328cb09c9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c211bb17b984b04b28cea328cb09c9b" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 41253 } }
I20260928 20:23:56.736400 25198 leader_election.cc:302] T 00000000000000000000000000000000 P 8c211bb17b984b04b28cea328cb09c9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c211bb17b984b04b28cea328cb09c9b; no voters:
I20260928 20:23:56.736588 25198 leader_election.cc:288] T 00000000000000000000000000000000 P 8c211bb17b984b04b28cea328cb09c9b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:23:56.736617 25203 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c211bb17b984b04b28cea328cb09c9b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:23:56.736766 25203 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c211bb17b984b04b28cea328cb09c9b [term 1 LEADER]: Becoming Leader. State: Replica: 8c211bb17b984b04b28cea328cb09c9b, State: Running, Role: LEADER
I20260928 20:23:56.736898 25198 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c211bb17b984b04b28cea328cb09c9b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:23:56.736928 25203 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c211bb17b984b04b28cea328cb09c9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c211bb17b984b04b28cea328cb09c9b" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 41253 } }
I20260928 20:23:56.737313 25203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c211bb17b984b04b28cea328cb09c9b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c211bb17b984b04b28cea328cb09c9b. Latest consensus state: current_term: 1 leader_uuid: "8c211bb17b984b04b28cea328cb09c9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c211bb17b984b04b28cea328cb09c9b" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 41253 } } }
I20260928 20:23:56.737537 25203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c211bb17b984b04b28cea328cb09c9b [sys.catalog]: This master's current role is: LEADER
I20260928 20:23:56.737458 25204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c211bb17b984b04b28cea328cb09c9b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c211bb17b984b04b28cea328cb09c9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c211bb17b984b04b28cea328cb09c9b" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 41253 } } }
I20260928 20:23:56.737776 25204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c211bb17b984b04b28cea328cb09c9b [sys.catalog]: This master's current role is: LEADER
I20260928 20:23:56.738047 25212 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:23:56.738610 25212 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:23:56.740572 25212 catalog_manager.cc:1347] Generated new cluster ID: a08d667eb0ea4305ae73658cf0c6fdf2
I20260928 20:23:56.740628 25212 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:23:56.742060 25202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:56.736176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40157 (local address 127.3.18.62:41253)
0928 20:23:56.736361 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:56.736372 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:56.736411 (+ 39us) server_negotiation.cc:407] Connection header received
0928 20:23:56.736546 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:56.736555 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:56.736636 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:56.736809 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:56.737223 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:56.738854 (+ 1631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:56.740035 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:56.740510 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:56.740636 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:56.741388 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:56.741391 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:56.741392 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:23:56.741590 (+ 198us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:56.741624 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:23:56.741636 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:56.741715 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:56.741793 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 20:23:56.741863 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":94,"threads_started":1}
I20260928 20:23:56.744971 3144 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 20:23:56.751006 25202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:56.746599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39287 (local address 127.3.18.62:41253)
0928 20:23:56.746646 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:56.746648 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:56.747963 (+ 1315us) server_negotiation.cc:407] Connection header received
0928 20:23:56.748041 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:56.748043 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:56.748055 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:56.748113 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:56.748344 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:56.748854 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:56.749686 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:56.749877 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:56.749959 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:56.750720 (+ 761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:56.750722 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:56.750722 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:23:56.750729 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:56.750745 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:23:56.750751 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:56.750802 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:56.750867 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 20:23:56.750896 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 20:23:56.753156 25212 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:23:56.753805 25212 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:23:56.760466 25212 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c211bb17b984b04b28cea328cb09c9b: Generated new TSK 0
I20260928 20:23:56.760725 25212 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 20:23:56.766680 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 25130
2026-09-28T20:23:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1328 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790626764250411-3144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:23:56 dist-test-slave-b83v krb5kdc[25229](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:23:56 dist-test-slave-b83v krb5kdc[25229](info): set up 2 sockets
Sep 28 20:23:56 dist-test-slave-b83v krb5kdc[25229](info): commencing operation
krb5kdc: starting...
W20260928 20:23:57.820331 3144 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Sep 28 20:23:57 dist-test-slave-b83v krb5kdc[25229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:23:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:23:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.18.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab.
Entry for principal kudu/127.3.18.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab.
I20260928 20:23:57.947523 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790626764250411-3144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790626764250411-3144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790626764250411-3144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:36463
--webserver_interface=127.3.18.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:44291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.18.62:36463
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:23:58.030637 25242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:23:58.030893 25242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:23:58.030933 25242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:23:58.032119 25242 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:23:58.032193 25242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:23:58.032218 25242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:23:58.032239 25242 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:23:58.032269 25242 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:23:58.033751 25242 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:44291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790626764250411-3144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790626764250411-3144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.18.62:36463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:36463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.18.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790626764250411-3144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:23:58.034050 25242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:23:58.034289 25242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:23:58.039445 25250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:58.039469 25248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:58.039587 25247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:23:58.039753 25242 server_base.cc:1034] running on GCE node
I20260928 20:23:58.040076 25242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:23:58.040352 25242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:23:58.041550 25242 hybrid_clock.cc:648] HybridClock initialized: now 1790627038041534 us; error 34 us; skew 500 ppm
Sep 28 20:23:58 dist-test-slave-b83v krb5kdc[25229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627038, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:23:58.044440 25242 init.cc:377] Logged in from keytab as kudu/127.3.18.62@KRBTEST.COM (short username kudu)
I20260928 20:23:58.044921 25242 webserver.cc:460] Webserver started at http://127.3.18.62:33461/ using document root <none> and password file <none>
I20260928 20:23:58.045166 25242 fs_manager.cc:359] Metadata directory not provided
I20260928 20:23:58.045235 25242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:23:58.045388 25242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:23:58.046463 25242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790626764250411-3144-0/minicluster-data/master-0/data/instance:
uuid: "65cc25e468f54b7f80495ab29cb394ed"
format_stamp: "Formatted at 2026-09-28 20:23:58 on dist-test-slave-b83v"
server_key: "9e2e487ca40f738d7b24936511e8dacb"
server_key_iv: "76559315017eead64850ec31cdbd484b"
server_key_version: "encryptionkey@0"
I20260928 20:23:58.046837 25242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790626764250411-3144-0/minicluster-data/master-0/wal/instance:
uuid: "65cc25e468f54b7f80495ab29cb394ed"
format_stamp: "Formatted at 2026-09-28 20:23:58 on dist-test-slave-b83v"
server_key: "9e2e487ca40f738d7b24936511e8dacb"
server_key_iv: "76559315017eead64850ec31cdbd484b"
server_key_version: "encryptionkey@0"
I20260928 20:23:58.048262 25242 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:23:58.049008 25257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:58.049235 25242 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:23:58.049345 25242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790626764250411-3144-0/minicluster-data/master-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790626764250411-3144-0/minicluster-data/master-0/wal
uuid: "65cc25e468f54b7f80495ab29cb394ed"
format_stamp: "Formatted at 2026-09-28 20:23:58 on dist-test-slave-b83v"
server_key: "9e2e487ca40f738d7b24936511e8dacb"
server_key_iv: "76559315017eead64850ec31cdbd484b"
server_key_version: "encryptionkey@0"
I20260928 20:23:58.049422 25242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790626764250411-3144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790626764250411-3144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790626764250411-3144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:23:58.052956 25242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:23:58.055300 25242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:23:58.055461 25242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:23:58.059479 25242 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.62:36463
I20260928 20:23:58.059657 25309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.62:36463 every 8 connection(s)
I20260928 20:23:58.059930 25242 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
I20260928 20:23:58.060921 25310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:23:58.062827 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 25242
I20260928 20:23:58.062958 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790626764250411-3144-0/minicluster-data/master-0/wal/instance
I20260928 20:23:58.063122 3144 external_mini_cluster.cc:1428] Setting key b40462568e2559a7510eb94f3bc2f0e1
I20260928 20:23:58.063725 25310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65cc25e468f54b7f80495ab29cb394ed: Bootstrap starting.
I20260928 20:23:58.064383 25310 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65cc25e468f54b7f80495ab29cb394ed: Neither blocks nor log segments found. Creating new log.
I20260928 20:23:58.064683 25310 log.cc:824] T 00000000000000000000000000000000 P 65cc25e468f54b7f80495ab29cb394ed: Log is configured to *not* fsync() on all Append() calls
I20260928 20:23:58.065480 25310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65cc25e468f54b7f80495ab29cb394ed: No bootstrap required, opened a new log
I20260928 20:23:58.065987 25313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:58.063726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35535 (local address 127.3.18.62:36463)
0928 20:23:58.063976 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:58.063986 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:58.064022 (+ 36us) server_negotiation.cc:407] Connection header received
0928 20:23:58.064503 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:58.064512 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:58.064599 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:58.064726 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:58.065525 (+ 799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:58.065529 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:58.065530 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:23:58.065535 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:58.065603 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:23:58.065619 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:58.065681 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:58.065767 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 20:23:58.065812 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":162,"threads_started":1}
I20260928 20:23:58.067034 3144 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 20:23:58.067919 25310 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65cc25e468f54b7f80495ab29cb394ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65cc25e468f54b7f80495ab29cb394ed" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36463 } }
I20260928 20:23:58.068102 25310 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65cc25e468f54b7f80495ab29cb394ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:23:58.068217 25310 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65cc25e468f54b7f80495ab29cb394ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65cc25e468f54b7f80495ab29cb394ed, State: Initialized, Role: FOLLOWER
I20260928 20:23:58.068384 25310 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65cc25e468f54b7f80495ab29cb394ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65cc25e468f54b7f80495ab29cb394ed" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36463 } }
I20260928 20:23:58.068490 25310 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65cc25e468f54b7f80495ab29cb394ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:23:58.068536 25310 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65cc25e468f54b7f80495ab29cb394ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:23:58.068608 25310 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65cc25e468f54b7f80495ab29cb394ed [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:23:58.069391 25310 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65cc25e468f54b7f80495ab29cb394ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65cc25e468f54b7f80495ab29cb394ed" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36463 } }
I20260928 20:23:58.069584 25310 leader_election.cc:302] T 00000000000000000000000000000000 P 65cc25e468f54b7f80495ab29cb394ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65cc25e468f54b7f80495ab29cb394ed; no voters:
I20260928 20:23:58.069891 25310 leader_election.cc:288] T 00000000000000000000000000000000 P 65cc25e468f54b7f80495ab29cb394ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:23:58.069917 25320 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65cc25e468f54b7f80495ab29cb394ed [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:23:58.070214 25310 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65cc25e468f54b7f80495ab29cb394ed [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:23:58.070535 25320 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65cc25e468f54b7f80495ab29cb394ed [term 1 LEADER]: Becoming Leader. State: Replica: 65cc25e468f54b7f80495ab29cb394ed, State: Running, Role: LEADER
I20260928 20:23:58.070665 25320 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65cc25e468f54b7f80495ab29cb394ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65cc25e468f54b7f80495ab29cb394ed" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36463 } }
I20260928 20:23:58.070787 25313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:58.068676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53407 (local address 127.3.18.62:36463)
0928 20:23:58.069636 (+ 960us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:58.069638 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:58.069647 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:23:58.069669 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:58.069670 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:58.069674 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:58.069733 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:58.070437 (+ 704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:58.070438 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:58.070438 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:23:58.070443 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:58.070459 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:23:58.070464 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:58.070514 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:58.070645 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 20:23:58.070660 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":927}
I20260928 20:23:58.071158 25322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65cc25e468f54b7f80495ab29cb394ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65cc25e468f54b7f80495ab29cb394ed. Latest consensus state: current_term: 1 leader_uuid: "65cc25e468f54b7f80495ab29cb394ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65cc25e468f54b7f80495ab29cb394ed" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36463 } } }
I20260928 20:23:58.071323 25321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65cc25e468f54b7f80495ab29cb394ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65cc25e468f54b7f80495ab29cb394ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65cc25e468f54b7f80495ab29cb394ed" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36463 } } }
I20260928 20:23:58.071372 25322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65cc25e468f54b7f80495ab29cb394ed [sys.catalog]: This master's current role is: LEADER
I20260928 20:23:58.071405 25321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65cc25e468f54b7f80495ab29cb394ed [sys.catalog]: This master's current role is: LEADER
W20260928 20:23:58.072371 25335 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 65cc25e468f54b7f80495ab29cb394ed: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 20:23:58.072451 25335 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 20:23:58.072513 25332 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:23:58.072667 25332 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:23:58.077024 25332 catalog_manager.cc:1347] Generated new cluster ID: 02eda731b2a14f0aa197bdff5e384bd1
I20260928 20:23:58.077092 25332 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:23:58.086380 25332 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:23:58.086567 25332 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:23:58.094801 25332 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65cc25e468f54b7f80495ab29cb394ed: Generated new TSK 0
I20260928 20:23:58.095054 25332 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 20:23:58.097692 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 25242
2026-09-28T20:23:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1328 ms)
[----------] 18 tests from AuthTokenIssuingTest (24484 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:23:58 dist-test-slave-b83v krb5kdc[25341](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:23:58 dist-test-slave-b83v krb5kdc[25341](info): set up 2 sockets
Sep 28 20:23:58 dist-test-slave-b83v krb5kdc[25341](info): commencing operation
krb5kdc: starting...
W20260928 20:23:59.140247 3144 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Sep 28 20:23:59 dist-test-slave-b83v krb5kdc[25341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:23:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:23:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.18.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab.
Entry for principal kudu/127.3.18.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab.
I20260928 20:23:59.287916 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:36647
--webserver_interface=127.3.18.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:44129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.18.62:36647,127.3.18.61:44471,127.3.18.60:41701
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:23:59.373776 25354 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 20:23:59.374094 25354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:23:59.374126 25354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:23:59.374164 25354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:23:59.375561 25354 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:23:59.375666 25354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:23:59.375692 25354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:23:59.375713 25354 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:23:59.375742 25354 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:23:59.377544 25354 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:44129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.18.62:36647,127.3.18.61:44471,127.3.18.60:41701
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:36647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.18.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:23:59.377892 25354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:23:59.378149 25354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:23:59.383723 25360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:59.383718 25359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:59.383724 25362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:23:59.383800 25354 server_base.cc:1034] running on GCE node
I20260928 20:23:59.384342 25354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:23:59.384709 25354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:23:59.385924 25354 hybrid_clock.cc:648] HybridClock initialized: now 1790627039385934 us; error 127 us; skew 500 ppm
Sep 28 20:23:59 dist-test-slave-b83v krb5kdc[25341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627039, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:23:59.394150 25354 init.cc:377] Logged in from keytab as kudu/127.3.18.62@KRBTEST.COM (short username kudu)
I20260928 20:23:59.394827 25354 webserver.cc:460] Webserver started at http://127.3.18.62:40313/ using document root <none> and password file <none>
I20260928 20:23:59.395143 25354 fs_manager.cc:359] Metadata directory not provided
I20260928 20:23:59.395313 25354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:23:59.395501 25354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:23:59.396843 25354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/minicluster-data/master-0/data/instance:
uuid: "d3fde98bbb314312a92222889171dde1"
format_stamp: "Formatted at 2026-09-28 20:23:59 on dist-test-slave-b83v"
server_key: "9f0066d6a7509b62dc7e8eeaeef94f88"
server_key_iv: "25dbfd88224761fdc9f61ee33a53c908"
server_key_version: "encryptionkey@0"
I20260928 20:23:59.397236 25354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/minicluster-data/master-0/wal/instance:
uuid: "d3fde98bbb314312a92222889171dde1"
format_stamp: "Formatted at 2026-09-28 20:23:59 on dist-test-slave-b83v"
server_key: "9f0066d6a7509b62dc7e8eeaeef94f88"
server_key_iv: "25dbfd88224761fdc9f61ee33a53c908"
server_key_version: "encryptionkey@0"
I20260928 20:23:59.398696 25354 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:23:59.399870 25369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:59.400146 25354 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:23:59.400278 25354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/minicluster-data/master-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/minicluster-data/master-0/wal
uuid: "d3fde98bbb314312a92222889171dde1"
format_stamp: "Formatted at 2026-09-28 20:23:59 on dist-test-slave-b83v"
server_key: "9f0066d6a7509b62dc7e8eeaeef94f88"
server_key_iv: "25dbfd88224761fdc9f61ee33a53c908"
server_key_version: "encryptionkey@0"
I20260928 20:23:59.400373 25354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:23:59.413374 25354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:23:59.415987 25354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:23:59.416169 25354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:23:59.420423 25354 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.62:36647
I20260928 20:23:59.420496 25421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.62:36647 every 8 connection(s)
I20260928 20:23:59.420849 25354 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
I20260928 20:23:59.421818 25422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:23:59.423465 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 25354
I20260928 20:23:59.423403 25422 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36647 } has no permanent_uuid. Determining permanent_uuid...
I20260928 20:23:59.423626 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/minicluster-data/master-0/wal/instance
I20260928 20:23:59.423878 3144 external_mini_cluster.cc:1428] Setting key b52a4cfc8d7ab148f654a4c0c4d365a2
Sep 28 20:23:59 dist-test-slave-b83v krb5kdc[25341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627039, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.62@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.18.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.61@KRBTEST.COM" created.
I20260928 20:23:59.435226 25425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:59.424418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39244 (local address 127.3.18.62:36647)
0928 20:23:59.424579 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:59.424584 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:59.424627 (+ 43us) server_negotiation.cc:407] Connection header received
0928 20:23:59.425107 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:59.425111 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:59.425136 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:59.425242 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:59.426032 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:59.426859 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:59.428372 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:59.428709 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:59.428801 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:59.431167 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:59.431175 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:59.431176 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:23:59.431183 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:59.433276 (+ 2093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:59.433905 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:59.433908 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:59.433908 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:59.433990 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:59.434184 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:59.434184 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:59.434184 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:59.434447 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:59.434542 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:59.434807 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 20:23:59.434964 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":62,"threads_started":1}
I20260928 20:23:59.435225 25424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:59.424209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36647 (local address 127.0.0.1:39244)
0928 20:23:59.424380 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:59.424424 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 20:23:59.424917 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:23:59.425272 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:23:59.425275 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:23:59.425672 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:23:59.425918 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:59.425925 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:59.426973 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:59.426975 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:59.428267 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:59.428271 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:59.428823 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:59.428824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:59.428922 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:59.429473 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:23:59.429485 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:23:59.431080 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:23:59.433357 (+ 2277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:23:59.433365 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:23:59.433368 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:23:59.433803 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:23:59.434054 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:23:59.434054 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:23:59.434055 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:23:59.434143 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:23:59.434531 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:23:59.434536 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:23:59.434677 (+ 141us) client_negotiation.cc:769] Sending connection context
0928 20:23:59.434781 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 20:23:59.435029 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":96,"thread_start_us":92,"threads_started":1}
I20260928 20:23:59.436308 25422 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.18.61" port: 44471 } has no permanent_uuid. Determining permanent_uuid...
I20260928 20:23:59.436776 25424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:59.436630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.61:44471 (local address 127.0.0.1:44028)
0928 20:23:59.436687 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:59.436743 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.61:44471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 20:23:59.437029 25371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.18.61:44471: connect: Connection refused (error 111)
I20260928 20:23:59.437522 25424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:59.437421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.61:44471 (local address 127.0.0.1:44044)
0928 20:23:59.437473 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:59.437498 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.61:44471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 20:23:59.437747 25422 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.18.61:44471: Network error: Client connection negotiation failed: client connection to 127.3.18.61:44471: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.61.keytab.
Entry for principal kudu/127.3.18.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.61.keytab.
I20260928 20:23:59.452167 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.61:44471
--webserver_interface=127.3.18.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.61.keytab
--principal=kudu/127.3.18.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:44129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.18.62:36647,127.3.18.61:44471,127.3.18.60:41701
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 20:23:59.486992 25422 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.18.61" port: 44471 } attempt: 1
I20260928 20:23:59.487615 25424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:59.487462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.61:44471 (local address 127.0.0.1:44048)
0928 20:23:59.487528 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:59.487579 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.61:44471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 20:23:59.487990 25424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:59.487906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.61:44471 (local address 127.0.0.1:44052)
0928 20:23:59.487937 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:59.487966 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.61:44471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260928 20:23:59.488163 25422 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.18.61:44471: Network error: Client connection negotiation failed: client connection to 127.3.18.61:44471: connect: Connection refused (error 111)
W20260928 20:23:59.536020 25429 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 20:23:59.536253 25429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:23:59.536273 25429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:23:59.536295 25429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:23:59.537557 25429 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:23:59.537621 25429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:23:59.537634 25429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:23:59.537647 25429 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:23:59.537660 25429 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:23:59.539247 25429 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:44129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.18.62:36647,127.3.18.61:44471,127.3.18.60:41701
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.61.keytab
--principal=kudu/127.3.18.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.61:44471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.18.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:23:59.539526 25429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:23:59.539805 25429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 20:23:59.545204 25429 server_base.cc:1034] running on GCE node
W20260928 20:23:59.545182 25434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:59.545313 25437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:59.545163 25435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:23:59.545766 25429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:23:59.546113 25429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:23:59.547263 25429 hybrid_clock.cc:648] HybridClock initialized: now 1790627039547237 us; error 40 us; skew 500 ppm
Sep 28 20:23:59 dist-test-slave-b83v krb5kdc[25341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627039, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:23:59.550110 25429 init.cc:377] Logged in from keytab as kudu/127.3.18.61@KRBTEST.COM (short username kudu)
I20260928 20:23:59.550665 25429 webserver.cc:460] Webserver started at http://127.3.18.61:38127/ using document root <none> and password file <none>
I20260928 20:23:59.550936 25429 fs_manager.cc:359] Metadata directory not provided
I20260928 20:23:59.550984 25429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:23:59.551158 25429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:23:59.552387 25429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/minicluster-data/master-1/data/instance:
uuid: "d06a1f0a4eae436ab1219f5797104e14"
format_stamp: "Formatted at 2026-09-28 20:23:59 on dist-test-slave-b83v"
server_key: "47acf091d6c5f4e6b4d6ca050a2b8497"
server_key_iv: "d77157d7939c9ed28c1b92c7d9ac9285"
server_key_version: "encryptionkey@0"
I20260928 20:23:59.552778 25429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/minicluster-data/master-1/wal/instance:
uuid: "d06a1f0a4eae436ab1219f5797104e14"
format_stamp: "Formatted at 2026-09-28 20:23:59 on dist-test-slave-b83v"
server_key: "47acf091d6c5f4e6b4d6ca050a2b8497"
server_key_iv: "d77157d7939c9ed28c1b92c7d9ac9285"
server_key_version: "encryptionkey@0"
I20260928 20:23:59.554291 25429 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 20:23:59.555262 25444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:59.555474 25429 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:23:59.555636 25429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/minicluster-data/master-1/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/minicluster-data/master-1/wal
uuid: "d06a1f0a4eae436ab1219f5797104e14"
format_stamp: "Formatted at 2026-09-28 20:23:59 on dist-test-slave-b83v"
server_key: "47acf091d6c5f4e6b4d6ca050a2b8497"
server_key_iv: "d77157d7939c9ed28c1b92c7d9ac9285"
server_key_version: "encryptionkey@0"
I20260928 20:23:59.555728 25429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:23:59.556347 25422 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.18.61" port: 44471 } attempt: 2
I20260928 20:23:59.556799 25424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:59.556661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.61:44471 (local address 127.0.0.1:44062)
0928 20:23:59.556724 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:59.556760 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.61:44471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 20:23:59.557140 25424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:59.557035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.61:44471 (local address 127.0.0.1:44068)
0928 20:23:59.557085 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:59.557109 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.61:44471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 20:23:59.557230 25422 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.18.61:44471: Network error: Client connection negotiation failed: client connection to 127.3.18.61:44471: connect: Connection refused (error 111)
I20260928 20:23:59.567294 25429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:23:59.569818 25429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:23:59.569989 25429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:23:59.574121 25429 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.61:44471
I20260928 20:23:59.574160 25496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.61:44471 every 8 connection(s)
I20260928 20:23:59.574621 25429 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/minicluster-data/master-1/data/info.pb
I20260928 20:23:59.575824 25497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:23:59.577550 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 25429
I20260928 20:23:59.577400 25497 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36647 } has no permanent_uuid. Determining permanent_uuid...
I20260928 20:23:59.577687 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/minicluster-data/master-1/wal/instance
I20260928 20:23:59.577914 3144 external_mini_cluster.cc:1428] Setting key 6d86dabbfcefdecc9efce02f2001aebd
Sep 28 20:23:59 dist-test-slave-b83v krb5kdc[25341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627039, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.61@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.18.60@KRBTEST.COM; defaulting to no policy
I20260928 20:23:59.587433 25425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:59.578295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39252 (local address 127.3.18.62:36647)
0928 20:23:59.578344 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:59.578346 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:59.578573 (+ 227us) server_negotiation.cc:407] Connection header received
0928 20:23:59.579030 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:59.579032 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:59.579048 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:59.579126 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:59.580073 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:59.580775 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:59.581766 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:59.582125 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:59.582206 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:59.584479 (+ 2273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:59.584481 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:59.584481 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:23:59.584489 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:59.586102 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:59.586521 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:59.586522 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:59.586522 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:59.586573 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:59.586797 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:59.586798 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:59.586798 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:59.586900 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:59.586952 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:59.587249 (+ 297us) server_negotiation.cc:299] Negotiation successful
0928 20:23:59.587336 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 20:23:59.587639 25499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:59.578269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36647 (local address 127.0.0.1:39252)
0928 20:23:59.578433 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:59.578476 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 20:23:59.578935 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:23:59.579226 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:23:59.579231 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:23:59.579625 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:23:59.579951 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:59.579959 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:59.580877 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:59.580877 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:59.581678 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:59.581680 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:59.582227 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:59.582228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:59.582337 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:59.582921 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:23:59.582930 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:23:59.584382 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:23:59.586221 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:23:59.586228 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:23:59.586230 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:23:59.586442 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:23:59.586644 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:23:59.586644 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:23:59.586645 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:23:59.586739 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:23:59.586957 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:23:59.586962 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:23:59.587078 (+ 116us) client_negotiation.cc:769] Sending connection context
0928 20:23:59.587224 (+ 146us) client_negotiation.cc:240] Negotiation successful
0928 20:23:59.587379 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":86,"threads_started":1}
I20260928 20:23:59.588274 25497 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.18.61" port: 44471 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.60@KRBTEST.COM" created.
Sep 28 20:23:59 dist-test-slave-b83v krb5kdc[25341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627039, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.61@KRBTEST.COM for kudu/127.3.18.61@KRBTEST.COM
I20260928 20:23:59.597102 25499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:59.588536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.61:44471 (local address 127.0.0.1:44084)
0928 20:23:59.588585 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:59.588595 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 20:23:59.588638 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:23:59.589094 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:23:59.589095 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:23:59.589261 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:23:59.589318 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:59.589319 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:59.590592 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:59.590594 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:59.591452 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:59.591455 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:59.592073 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:59.592074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:59.592158 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:59.592564 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:23:59.592570 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:23:59.593918 (+ 1348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:23:59.595951 (+ 2033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:23:59.595952 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:23:59.595953 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:23:59.596131 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:23:59.596363 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:23:59.596364 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:23:59.596364 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:23:59.596420 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:23:59.596764 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:23:59.596764 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:23:59.596812 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 20:23:59.596877 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 20:23:59.596984 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 20:23:59.597128 25501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:59.588738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44084 (local address 127.3.18.61:44471)
0928 20:23:59.588948 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:59.588953 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:59.588977 (+ 24us) server_negotiation.cc:407] Connection header received
0928 20:23:59.588993 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:59.588994 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:59.589004 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:59.589060 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:59.589405 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:59.590492 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:59.591542 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:59.591960 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:59.592020 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:59.594016 (+ 1996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:59.594026 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:59.594026 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:23:59.594033 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:59.595856 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:59.596239 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:59.596242 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:59.596243 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:59.596298 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:59.596487 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:59.596488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:59.596488 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:59.596687 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:59.596791 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:59.596911 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 20:23:59.597036 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":164,"thread_start_us":113,"threads_started":1}
I20260928 20:23:59.597874 25497 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.18.60" port: 41701 } has no permanent_uuid. Determining permanent_uuid...
I20260928 20:23:59.598383 25499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:59.598272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.60:41701 (local address 127.0.0.1:46020)
0928 20:23:59.598319 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:59.598363 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.60:41701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 20:23:59.598598 25447 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.18.60:41701: connect: Connection refused (error 111)
I20260928 20:23:59.599058 25499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:59.598960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.60:41701 (local address 127.0.0.1:46028)
0928 20:23:59.599015 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:59.599040 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.60:41701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 20:23:59.599175 25497 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.18.60:41701: Network error: Client connection negotiation failed: client connection to 127.3.18.60:41701: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.60.keytab.
Entry for principal kudu/127.3.18.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.60.keytab.
I20260928 20:23:59.607082 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.60:41701
--webserver_interface=127.3.18.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.60.keytab
--principal=kudu/127.3.18.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:44129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.18.62:36647,127.3.18.61:44471,127.3.18.60:41701
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 20:23:59.648389 25422 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.18.61" port: 44471 } attempt: 3
I20260928 20:23:59.648413 25497 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.18.60" port: 41701 } attempt: 1
I20260928 20:23:59.648941 25499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:59.648743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.60:41701 (local address 127.0.0.1:46032)
0928 20:23:59.648861 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:59.648898 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.60:41701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 20:23:59.649322 25499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:59.649217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.60:41701 (local address 127.0.0.1:46048)
0928 20:23:59.649267 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:59.649301 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.60:41701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 20:23:59.649415 25497 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.18.60:41701: Network error: Client connection negotiation failed: client connection to 127.3.18.60:41701: connect: Connection refused (error 111)
Sep 28 20:23:59 dist-test-slave-b83v krb5kdc[25341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627039, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.62@KRBTEST.COM for kudu/127.3.18.61@KRBTEST.COM
I20260928 20:23:59.656457 25424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:59.648767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.61:44471 (local address 127.0.0.1:44098)
0928 20:23:59.648818 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:59.648832 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 20:23:59.648879 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:23:59.649074 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:23:59.649075 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:23:59.649276 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:23:59.649359 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:59.649361 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:59.650354 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:59.650355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:59.650893 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:59.650895 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:59.651414 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:59.651415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:59.651504 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:59.652090 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:23:59.652095 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:23:59.653355 (+ 1260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:23:59.655376 (+ 2021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:23:59.655377 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:23:59.655377 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:23:59.655570 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:23:59.655845 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:23:59.655846 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:23:59.655847 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:23:59.655905 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:23:59.656192 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:23:59.656193 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:23:59.656240 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 20:23:59.656295 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 20:23:59.656383 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 20:23:59.656518 25501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:59.648827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44098 (local address 127.3.18.61:44471)
0928 20:23:59.648890 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:59.648892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:59.648905 (+ 13us) server_negotiation.cc:407] Connection header received
0928 20:23:59.648938 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:59.648939 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:59.648951 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:59.649436 (+ 485us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:59.649475 (+ 39us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:59.650246 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:59.650982 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:59.651330 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:59.651406 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:59.653457 (+ 2051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:59.653459 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:59.653459 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:23:59.653468 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:59.655247 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:59.655651 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:59.655652 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:59.655653 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:59.655724 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:59.655973 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:59.655974 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:59.655974 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:59.656125 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:59.656192 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:59.656340 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 20:23:59.656426 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 20:23:59.657217 25422 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.18.60" port: 41701 } has no permanent_uuid. Determining permanent_uuid...
I20260928 20:23:59.657603 25424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:59.657491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.60:41701 (local address 127.0.0.1:46064)
0928 20:23:59.657541 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:59.657574 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.60:41701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 20:23:59.657917 25424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:59.657817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.60:41701 (local address 127.0.0.1:46066)
0928 20:23:59.657865 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:59.657888 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.60:41701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 20:23:59.658003 25422 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.18.60:41701: Network error: Client connection negotiation failed: client connection to 127.3.18.60:41701: connect: Connection refused (error 111)
I20260928 20:23:59.689189 25422 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.18.60" port: 41701 } attempt: 1
I20260928 20:23:59.689709 25424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:59.689563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.60:41701 (local address 127.0.0.1:46082)
0928 20:23:59.689624 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:59.689666 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.60:41701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 20:23:59.690122 25424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:59.690026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.60:41701 (local address 127.0.0.1:46094)
0928 20:23:59.690077 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:59.690101 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.60:41701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 20:23:59.690217 25422 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.18.60:41701: Network error: Client connection negotiation failed: client connection to 127.3.18.60:41701: connect: Connection refused (error 111)
W20260928 20:23:59.694329 25504 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 20:23:59.694540 25504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:23:59.694558 25504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:23:59.694619 25504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:23:59.695942 25504 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:23:59.695986 25504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:23:59.696002 25504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:23:59.696014 25504 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:23:59.696058 25504 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:23:59.697598 25504 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:44129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.18.62:36647,127.3.18.61:44471,127.3.18.60:41701
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.60.keytab
--principal=kudu/127.3.18.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.60:41701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.18.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:23:59.697865 25504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:23:59.698123 25504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:23:59.703110 25512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:59.703181 25509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:23:59.703140 25510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:23:59.703850 25504 server_base.cc:1034] running on GCE node
I20260928 20:23:59.704214 25504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:23:59.704614 25504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:23:59.705802 25504 hybrid_clock.cc:648] HybridClock initialized: now 1790627039705783 us; error 29 us; skew 500 ppm
Sep 28 20:23:59 dist-test-slave-b83v krb5kdc[25341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627039, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:23:59.708426 25504 init.cc:377] Logged in from keytab as kudu/127.3.18.60@KRBTEST.COM (short username kudu)
I20260928 20:23:59.708896 25504 webserver.cc:460] Webserver started at http://127.3.18.60:37787/ using document root <none> and password file <none>
I20260928 20:23:59.709142 25504 fs_manager.cc:359] Metadata directory not provided
I20260928 20:23:59.709216 25504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:23:59.709360 25504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:23:59.710418 25504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/minicluster-data/master-2/data/instance:
uuid: "be945e9efd744a6eabb4f57b82602d7c"
format_stamp: "Formatted at 2026-09-28 20:23:59 on dist-test-slave-b83v"
server_key: "07b789017b36357e2eccdf42159ee6d8"
server_key_iv: "35edc5caa2decf0d610a872c69051020"
server_key_version: "encryptionkey@0"
I20260928 20:23:59.710814 25504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/minicluster-data/master-2/wal/instance:
uuid: "be945e9efd744a6eabb4f57b82602d7c"
format_stamp: "Formatted at 2026-09-28 20:23:59 on dist-test-slave-b83v"
server_key: "07b789017b36357e2eccdf42159ee6d8"
server_key_iv: "35edc5caa2decf0d610a872c69051020"
server_key_version: "encryptionkey@0"
I20260928 20:23:59.712244 25504 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:23:59.713045 25519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:23:59.713248 25504 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:23:59.713352 25504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/minicluster-data/master-2/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/minicluster-data/master-2/wal
uuid: "be945e9efd744a6eabb4f57b82602d7c"
format_stamp: "Formatted at 2026-09-28 20:23:59 on dist-test-slave-b83v"
server_key: "07b789017b36357e2eccdf42159ee6d8"
server_key_iv: "35edc5caa2decf0d610a872c69051020"
server_key_version: "encryptionkey@0"
I20260928 20:23:59.713426 25504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:23:59.717547 25497 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.18.60" port: 41701 } attempt: 2
I20260928 20:23:59.717957 25499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:59.717838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.60:41701 (local address 127.0.0.1:46096)
0928 20:23:59.717892 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:59.717929 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.60:41701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 20:23:59.718446 25499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:59.718280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.60:41701 (local address 127.0.0.1:46112)
0928 20:23:59.718329 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:59.718373 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.60:41701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 20:23:59.718582 25497 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.18.60:41701: Network error: Client connection negotiation failed: client connection to 127.3.18.60:41701: connect: Connection refused (error 111)
I20260928 20:23:59.728849 25504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:23:59.731160 25504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:23:59.731323 25504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:23:59.735502 25504 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.60:41701
I20260928 20:23:59.735636 25571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.60:41701 every 8 connection(s)
I20260928 20:23:59.735985 25504 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/minicluster-data/master-2/data/info.pb
I20260928 20:23:59.737051 25572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:23:59.738574 25572 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36647 } has no permanent_uuid. Determining permanent_uuid...
I20260928 20:23:59.742722 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 25504
I20260928 20:23:59.742839 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790626764250411-3144-0/minicluster-data/master-2/wal/instance
I20260928 20:23:59.743005 3144 external_mini_cluster.cc:1428] Setting key 2d9da32b511c1f5404e6f5683fb4ccf2
Sep 28 20:23:59 dist-test-slave-b83v krb5kdc[25341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627039, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.60@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:23:59.748582 25425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:59.739380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39254 (local address 127.3.18.62:36647)
0928 20:23:59.739462 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:59.739464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:59.739675 (+ 211us) server_negotiation.cc:407] Connection header received
0928 20:23:59.740331 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:59.740332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:59.740345 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:59.740410 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:59.741085 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:59.741587 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:59.742630 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:59.742827 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:59.742909 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:59.745349 (+ 2440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:59.745351 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:59.745351 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:23:59.745360 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:59.747041 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:59.747493 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:59.747494 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:59.747494 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:59.747564 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:59.747862 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:59.747863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:59.747864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:59.748002 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:59.748060 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:59.748440 (+ 380us) server_negotiation.cc:299] Negotiation successful
0928 20:23:59.748511 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 20:23:59.748868 25574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:59.739340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:36647 (local address 127.0.0.1:39254)
0928 20:23:59.739530 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:59.739583 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 20:23:59.740215 (+ 632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:23:59.740485 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:23:59.740489 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:23:59.740754 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:23:59.740953 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:59.740958 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:59.741690 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:59.741691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:59.742522 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:59.742524 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:59.742990 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:59.742991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:59.743141 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:59.743845 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:23:59.743855 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:23:59.745251 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:23:59.747122 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:23:59.747130 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:23:59.747133 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:23:59.747394 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:23:59.747631 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:23:59.747632 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:23:59.747632 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:23:59.747745 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:23:59.748077 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:23:59.748084 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:23:59.748281 (+ 197us) client_negotiation.cc:769] Sending connection context
0928 20:23:59.748406 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 20:23:59.748615 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":97,"threads_started":1}
I20260928 20:23:59.749447 25572 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.18.61" port: 44471 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 20:23:59 dist-test-slave-b83v krb5kdc[25341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.18.60@KRBTEST.COM
I20260928 20:23:59.753428 25576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:59.743463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46126 (local address 127.3.18.60:41701)
0928 20:23:59.743689 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:59.743694 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:59.744003 (+ 309us) server_negotiation.cc:407] Connection header received
0928 20:23:59.744067 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:59.744070 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:59.744097 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:59.744194 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:59.744819 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:59.745583 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:59.746792 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:59.747250 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:59.747339 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:59.750147 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:59.750157 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:59.750157 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:23:59.750164 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:59.752122 (+ 1958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:59.752494 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:59.752498 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:59.752498 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:59.752561 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:59.752830 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:59.752831 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:59.752831 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:59.753008 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:59.753175 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:59.753221 (+ 46us) server_negotiation.cc:299] Negotiation successful
0928 20:23:59.753299 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":95,"threads_started":1}
Sep 28 20:23:59 dist-test-slave-b83v krb5kdc[25341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627039, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.60@KRBTEST.COM for kudu/127.3.18.61@KRBTEST.COM
I20260928 20:23:59.756786 25574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:59.749728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.61:44471 (local address 127.0.0.1:44104)
0928 20:23:59.749778 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:59.749788 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 20:23:59.749827 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:23:59.749979 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:23:59.749979 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:23:59.750092 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:23:59.750174 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:59.750176 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:59.750812 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:59.750813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:59.751644 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:59.751647 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:59.752076 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:59.752078 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:59.752168 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:59.752592 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:23:59.752597 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:23:59.753915 (+ 1318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:23:59.755682 (+ 1767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:23:59.755683 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:23:59.755683 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:23:59.755921 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:23:59.756142 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:23:59.756143 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:23:59.756143 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:23:59.756202 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:23:59.756459 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:23:59.756460 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:23:59.756504 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 20:23:59.756591 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 20:23:59.756689 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 20:23:59.756786 25501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:59.749810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44104 (local address 127.3.18.61:44471)
0928 20:23:59.749857 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:59.749859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:59.749868 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:23:59.749896 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:59.749897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:59.749908 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:59.749977 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:59.750275 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:59.750732 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:59.751735 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:59.751999 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:59.752059 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:59.754001 (+ 1942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:59.754003 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:59.754003 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:23:59.754012 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:59.755563 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:59.756006 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:59.756008 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:59.756008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:59.756072 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:59.756274 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:59.756275 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:59.756275 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:59.756392 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:59.756468 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:59.756620 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 20:23:59.756697 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 20:23:59.757423 25572 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.18.60" port: 41701 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 20:23:59 dist-test-slave-b83v krb5kdc[25341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
Sep 28 20:23:59 dist-test-slave-b83v krb5kdc[25341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627039, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.60@KRBTEST.COM for kudu/127.3.18.60@KRBTEST.COM
I20260928 20:23:59.762619 25425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:59.754353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39262 (local address 127.3.18.62:36647)
0928 20:23:59.754394 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:59.754396 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:59.754403 (+ 7us) server_negotiation.cc:407] Connection header received
0928 20:23:59.754482 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:59.754483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:59.754494 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:59.754548 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:59.755037 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:59.755506 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:59.756268 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:59.756477 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:59.756562 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:59.759114 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:59.759116 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:59.759116 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:23:59.759123 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:59.760656 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:59.761032 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:59.761033 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:59.761034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:59.761117 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:59.761896 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:59.761897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:59.761897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:59.762077 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:59.762282 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:59.762387 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 20:23:59.762500 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 20:23:59.764928 25574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:59.757685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.60:41701 (local address 127.0.0.1:46130)
0928 20:23:59.757734 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:59.757744 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 20:23:59.757780 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:23:59.757923 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:23:59.757923 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:23:59.758055 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:23:59.758089 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:59.758090 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:59.758755 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:59.758756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:59.759507 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:59.759511 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:59.760169 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:59.760170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:59.760230 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:59.760616 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:23:59.760621 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:23:59.761945 (+ 1324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:23:59.763917 (+ 1972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:23:59.763918 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:23:59.763919 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:23:59.764112 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:23:59.764303 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:23:59.764303 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:23:59.764304 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:23:59.764369 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:23:59.764587 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:23:59.764588 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:23:59.764632 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 20:23:59.764703 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 20:23:59.764802 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 20:23:59.764986 25576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:59.757733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46130 (local address 127.3.18.60:41701)
0928 20:23:59.757839 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:59.757840 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:59.757844 (+ 4us) server_negotiation.cc:407] Connection header received
0928 20:23:59.757856 (+ 12us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:59.757856 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:59.757864 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:59.757919 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:59.758168 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:59.758690 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:59.759646 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:59.760032 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:59.760120 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:59.762038 (+ 1918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:59.762040 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:59.762048 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:23:59.762056 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:59.763841 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:59.764191 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:59.764192 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:59.764192 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:59.764242 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:59.764431 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:59.764431 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:59.764432 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:59.764529 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:59.764577 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:59.764730 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 20:23:59.764846 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":96}
I20260928 20:23:59.765376 25422 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.18.60" port: 41701 } attempt: 2
I20260928 20:23:59.766712 25572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be945e9efd744a6eabb4f57b82602d7c: Bootstrap starting.
I20260928 20:23:59.767861 25572 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be945e9efd744a6eabb4f57b82602d7c: Neither blocks nor log segments found. Creating new log.
I20260928 20:23:59.768230 25572 log.cc:824] T 00000000000000000000000000000000 P be945e9efd744a6eabb4f57b82602d7c: Log is configured to *not* fsync() on all Append() calls
Sep 28 20:23:59 dist-test-slave-b83v krb5kdc[25341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.18.61@KRBTEST.COM
I20260928 20:23:59.769155 25572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be945e9efd744a6eabb4f57b82602d7c: No bootstrap required, opened a new log
Sep 28 20:23:59 dist-test-slave-b83v krb5kdc[25341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627039, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.62@KRBTEST.COM for kudu/127.3.18.60@KRBTEST.COM
I20260928 20:23:59.771420 25572 raft_consensus.cc:348] T 00000000000000000000000000000000 P be945e9efd744a6eabb4f57b82602d7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3fde98bbb314312a92222889171dde1" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36647 } } peers { permanent_uuid: "d06a1f0a4eae436ab1219f5797104e14" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 44471 } } peers { permanent_uuid: "be945e9efd744a6eabb4f57b82602d7c" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 41701 } }
I20260928 20:23:59.771605 25572 raft_consensus.cc:729] T 00000000000000000000000000000000 P be945e9efd744a6eabb4f57b82602d7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be945e9efd744a6eabb4f57b82602d7c, State: Initialized, Role: FOLLOWER
I20260928 20:23:59.771703 25572 consensus_queue.cc:260] T 00000000000000000000000000000000 P be945e9efd744a6eabb4f57b82602d7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3fde98bbb314312a92222889171dde1" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36647 } } peers { permanent_uuid: "d06a1f0a4eae436ab1219f5797104e14" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 44471 } } peers { permanent_uuid: "be945e9efd744a6eabb4f57b82602d7c" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 41701 } }
I20260928 20:23:59.772064 25572 sys_catalog.cc:564] T 00000000000000000000000000000000 P be945e9efd744a6eabb4f57b82602d7c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:23:59.772300 25501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:59.763352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44118 (local address 127.3.18.61:44471)
0928 20:23:59.763390 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:59.763391 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:59.763400 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:23:59.763450 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:59.763451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:59.763458 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:59.763504 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:59.764127 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:59.764852 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:59.765842 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:59.766109 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:59.766155 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:59.769127 (+ 2972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:59.769128 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:59.769129 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:23:59.769137 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:59.770877 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:59.771297 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:59.771299 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:59.771299 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:59.771363 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:59.771591 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:59.771592 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:59.771593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:59.771747 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:59.771968 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:59.772113 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 20:23:59.772194 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 20:23:59.772696 25424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:59.765681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.60:41701 (local address 127.0.0.1:46132)
0928 20:23:59.765729 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:59.765741 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:23:59.765787 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:23:59.765919 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:23:59.765919 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:23:59.766072 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:23:59.766110 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:59.766112 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:59.766913 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:59.766914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:59.767439 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:59.767440 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:59.767869 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:59.767871 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:59.767935 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:59.768386 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:23:59.768391 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:23:59.769821 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:23:59.771460 (+ 1639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:23:59.771461 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:23:59.771461 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:23:59.771688 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:23:59.771930 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:23:59.771930 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:23:59.771931 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:23:59.771994 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:23:59.772360 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:23:59.772360 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:23:59.772411 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 20:23:59.772472 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 20:23:59.772592 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 20:23:59.772979 25576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:59.765713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46132 (local address 127.3.18.60:41701)
0928 20:23:59.765749 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:59.765750 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:59.765773 (+ 23us) server_negotiation.cc:407] Connection header received
0928 20:23:59.765843 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:59.765844 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:59.765853 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:59.765914 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:59.766193 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:59.766843 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:59.767532 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:59.767746 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:59.767867 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:59.769918 (+ 2051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:59.769919 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:59.769919 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:23:59.769926 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:59.771378 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:59.771769 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:59.771770 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:59.771771 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:59.771859 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:59.772059 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:59.772060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:59.772060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:59.772269 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:59.772330 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:59.772816 (+ 486us) server_negotiation.cc:299] Negotiation successful
0928 20:23:59.772892 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 20:23:59.773852 25578 sys_catalog.cc:455] T 00000000000000000000000000000000 P be945e9efd744a6eabb4f57b82602d7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3fde98bbb314312a92222889171dde1" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36647 } } peers { permanent_uuid: "d06a1f0a4eae436ab1219f5797104e14" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 44471 } } peers { permanent_uuid: "be945e9efd744a6eabb4f57b82602d7c" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 41701 } } }
I20260928 20:23:59.773939 25578 sys_catalog.cc:458] T 00000000000000000000000000000000 P be945e9efd744a6eabb4f57b82602d7c [sys.catalog]: This master's current role is: FOLLOWER
W20260928 20:23:59.774266 25589 catalog_manager.cc:1558] T 00000000000000000000000000000000 P be945e9efd744a6eabb4f57b82602d7c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 20:23:59.774333 25589 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 20:23:59.774528 3144 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 20:23:59.774991 25422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3fde98bbb314312a92222889171dde1: Bootstrap starting.
I20260928 20:23:59.775724 25422 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3fde98bbb314312a92222889171dde1: Neither blocks nor log segments found. Creating new log.
I20260928 20:23:59.776151 25422 log.cc:824] T 00000000000000000000000000000000 P d3fde98bbb314312a92222889171dde1: Log is configured to *not* fsync() on all Append() calls
I20260928 20:23:59.775651 25395 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d3fde98bbb314312a92222889171dde1"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39262
I20260928 20:23:59.776952 25422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3fde98bbb314312a92222889171dde1: No bootstrap required, opened a new log
I20260928 20:23:59.774580 25504 master_runner.cc:186] Error getting master registration for 127.3.18.62:36647: OK, instance_id { permanent_uuid: "d3fde98bbb314312a92222889171dde1" instance_seqno: 1790627039419171 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 20:23:59.777700 25504 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 20:23:59.778340 25422 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3fde98bbb314312a92222889171dde1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3fde98bbb314312a92222889171dde1" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36647 } } peers { permanent_uuid: "d06a1f0a4eae436ab1219f5797104e14" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 44471 } } peers { permanent_uuid: "be945e9efd744a6eabb4f57b82602d7c" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 41701 } }
I20260928 20:23:59.778496 25422 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3fde98bbb314312a92222889171dde1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3fde98bbb314312a92222889171dde1, State: Initialized, Role: FOLLOWER
I20260928 20:23:59.778596 25422 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3fde98bbb314312a92222889171dde1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3fde98bbb314312a92222889171dde1" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36647 } } peers { permanent_uuid: "d06a1f0a4eae436ab1219f5797104e14" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 44471 } } peers { permanent_uuid: "be945e9efd744a6eabb4f57b82602d7c" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 41701 } }
I20260928 20:23:59.778879 25592 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3fde98bbb314312a92222889171dde1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3fde98bbb314312a92222889171dde1" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36647 } } peers { permanent_uuid: "d06a1f0a4eae436ab1219f5797104e14" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 44471 } } peers { permanent_uuid: "be945e9efd744a6eabb4f57b82602d7c" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 41701 } } }
I20260928 20:23:59.778925 25422 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3fde98bbb314312a92222889171dde1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:23:59.778949 25395 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3fde98bbb314312a92222889171dde1 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 20:23:59.779006 25395 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3fde98bbb314312a92222889171dde1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:23:59.779954 25395 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3fde98bbb314312a92222889171dde1 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3fde98bbb314312a92222889171dde1" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36647 } } peers { permanent_uuid: "d06a1f0a4eae436ab1219f5797104e14" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 44471 } } peers { permanent_uuid: "be945e9efd744a6eabb4f57b82602d7c" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 41701 } }
I20260928 20:23:59.780316 25395 leader_election.cc:288] T 00000000000000000000000000000000 P d3fde98bbb314312a92222889171dde1 [CANDIDATE]: Term 1 election: Requested vote from peers d06a1f0a4eae436ab1219f5797104e14 (127.3.18.61:44471), be945e9efd744a6eabb4f57b82602d7c (127.3.18.60:41701)
I20260928 20:23:59.780666 25546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d3fde98bbb314312a92222889171dde1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "be945e9efd744a6eabb4f57b82602d7c"
I20260928 20:23:59.780793 25546 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be945e9efd744a6eabb4f57b82602d7c [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:23:59.781725 25546 raft_consensus.cc:2454] T 00000000000000000000000000000000 P be945e9efd744a6eabb4f57b82602d7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3fde98bbb314312a92222889171dde1 in term 1.
I20260928 20:23:59.782298 25372 leader_election.cc:302] T 00000000000000000000000000000000 P d3fde98bbb314312a92222889171dde1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be945e9efd744a6eabb4f57b82602d7c, d3fde98bbb314312a92222889171dde1; no voters:
I20260928 20:23:59.782644 25592 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3fde98bbb314312a92222889171dde1 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 20:23:59.782672 25609 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3fde98bbb314312a92222889171dde1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:23:59.782931 25609 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3fde98bbb314312a92222889171dde1 [term 1 LEADER]: Becoming Leader. State: Replica: d3fde98bbb314312a92222889171dde1, State: Running, Role: LEADER
I20260928 20:23:59.783401 25609 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3fde98bbb314312a92222889171dde1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3fde98bbb314312a92222889171dde1" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36647 } } peers { permanent_uuid: "d06a1f0a4eae436ab1219f5797104e14" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 44471 } } peers { permanent_uuid: "be945e9efd744a6eabb4f57b82602d7c" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 41701 } }
I20260928 20:23:59.785012 25592 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3fde98bbb314312a92222889171dde1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3fde98bbb314312a92222889171dde1. Latest consensus state: current_term: 1 leader_uuid: "d3fde98bbb314312a92222889171dde1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3fde98bbb314312a92222889171dde1" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36647 } } peers { permanent_uuid: "d06a1f0a4eae436ab1219f5797104e14" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 44471 } } peers { permanent_uuid: "be945e9efd744a6eabb4f57b82602d7c" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 41701 } } }
I20260928 20:23:59.785235 25592 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3fde98bbb314312a92222889171dde1 [sys.catalog]: This master's current role is: LEADER
I20260928 20:23:59.785559 25612 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:23:59.782898 25354 master_runner.cc:186] Error getting master registration for 127.3.18.61:44471: OK, instance_id { permanent_uuid: "d06a1f0a4eae436ab1219f5797104e14" instance_seqno: 1790627039572935 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 20:23:59.785969 25354 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 20:23:59.786139 25612 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:23:59.787657 25546 raft_consensus.cc:1261] T 00000000000000000000000000000000 P be945e9efd744a6eabb4f57b82602d7c [term 1 FOLLOWER]: Refusing update from remote peer d3fde98bbb314312a92222889171dde1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:23:59.788028 25609 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d3fde98bbb314312a92222889171dde1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be945e9efd744a6eabb4f57b82602d7c" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:23:59.789695 25578 sys_catalog.cc:455] T 00000000000000000000000000000000 P be945e9efd744a6eabb4f57b82602d7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3fde98bbb314312a92222889171dde1. Latest consensus state: current_term: 1 leader_uuid: "d3fde98bbb314312a92222889171dde1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3fde98bbb314312a92222889171dde1" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36647 } } peers { permanent_uuid: "d06a1f0a4eae436ab1219f5797104e14" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 44471 } } peers { permanent_uuid: "be945e9efd744a6eabb4f57b82602d7c" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 41701 } } }
I20260928 20:23:59.789811 25578 sys_catalog.cc:458] T 00000000000000000000000000000000 P be945e9efd744a6eabb4f57b82602d7c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 20:23:59.790580 25592 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3fde98bbb314312a92222889171dde1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d3fde98bbb314312a92222889171dde1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3fde98bbb314312a92222889171dde1" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36647 } } peers { permanent_uuid: "d06a1f0a4eae436ab1219f5797104e14" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 44471 } } peers { permanent_uuid: "be945e9efd744a6eabb4f57b82602d7c" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 41701 } } }
I20260928 20:23:59.790661 25592 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3fde98bbb314312a92222889171dde1 [sys.catalog]: This master's current role is: LEADER
I20260928 20:23:59.790896 25578 sys_catalog.cc:455] T 00000000000000000000000000000000 P be945e9efd744a6eabb4f57b82602d7c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d3fde98bbb314312a92222889171dde1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3fde98bbb314312a92222889171dde1" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36647 } } peers { permanent_uuid: "d06a1f0a4eae436ab1219f5797104e14" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 44471 } } peers { permanent_uuid: "be945e9efd744a6eabb4f57b82602d7c" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 41701 } } }
I20260928 20:23:59.790978 25578 sys_catalog.cc:458] T 00000000000000000000000000000000 P be945e9efd744a6eabb4f57b82602d7c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 20:23:59.791090 25614 mvcc.cc:205] Tried to move back new op lower bound from 7334408354967961600 to 7334408354956144640. Current Snapshot: MvccSnapshot[applied={T|T < 7334408354967961600}]
I20260928 20:23:59.792948 25615 mvcc.cc:205] Tried to move back new op lower bound from 7334408354967961600 to 7334408354956144640. Current Snapshot: MvccSnapshot[applied={T|T < 7334408354967961600}]
I20260928 20:23:59.793519 25612 catalog_manager.cc:1347] Generated new cluster ID: ae49c499b47844a08fe47b2a681181e4
I20260928 20:23:59.793581 25612 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:23:59.794653 25425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:59.782063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39264 (local address 127.3.18.62:36647)
0928 20:23:59.782212 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:59.782215 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:59.785676 (+ 3461us) server_negotiation.cc:407] Connection header received
0928 20:23:59.785882 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:59.785883 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:59.785895 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:59.785926 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:59.786910 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:59.787641 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:59.788828 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:59.789119 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:59.789196 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:59.790845 (+ 1649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:59.790846 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:59.790847 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:23:59.790854 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:59.792758 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:59.793455 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:59.793456 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:59.793457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:59.793534 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:59.793894 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:59.793895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:59.793895 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:59.794032 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:59.794104 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:59.794470 (+ 366us) server_negotiation.cc:299] Negotiation successful
0928 20:23:59.794554 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260928 20:23:59.801070 25576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:59.782406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46144 (local address 127.3.18.60:41701)
0928 20:23:59.782446 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:59.782449 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:59.792368 (+ 9919us) server_negotiation.cc:407] Connection header received
0928 20:23:59.792429 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:59.792430 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:59.792442 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:59.792501 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:59.793283 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:59.794002 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:59.795819 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:59.796163 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:59.796212 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:59.797763 (+ 1551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:59.797764 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:59.797764 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:23:59.797772 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:59.799485 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:59.799889 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:59.799890 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:59.799891 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:59.799954 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:59.800179 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:59.800180 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:59.800180 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:59.800579 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:59.800660 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:59.800881 (+ 221us) server_negotiation.cc:299] Negotiation successful
0928 20:23:59.800962 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 20:23:59.802620 25612 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:23:59.803308 25612 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:23:59.805956 25501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:59.791993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44122 (local address 127.3.18.61:44471)
0928 20:23:59.792042 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:59.792044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:59.795217 (+ 3173us) server_negotiation.cc:407] Connection header received
0928 20:23:59.795385 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:59.795386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:59.795414 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:59.795446 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:59.796743 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:59.797525 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:59.799361 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:59.799723 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:59.799818 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:59.802089 (+ 2271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:59.802090 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:59.802090 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:23:59.802098 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:59.803997 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:59.804540 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:59.804541 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:59.804541 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:59.804619 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:59.804868 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:59.804869 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:59.804869 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:59.805011 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:59.805554 (+ 543us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:59.805764 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 20:23:59.805858 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 20:23:59.807842 25612 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3fde98bbb314312a92222889171dde1: Generated new TSK 0
I20260928 20:23:59.808013 25612 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 20:23:59.809724 25497 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.18.60" port: 41701 } attempt: 3
Sep 28 20:23:59 dist-test-slave-b83v krb5kdc[25341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627039, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.61@KRBTEST.COM for kudu/127.3.18.60@KRBTEST.COM
I20260928 20:23:59.818653 25499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:59.809990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.60:41701 (local address 127.0.0.1:46148)
0928 20:23:59.810038 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 20:23:59.810048 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 20:23:59.810086 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:23:59.810245 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:23:59.810245 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:23:59.810404 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:23:59.810458 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:59.810459 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:59.811328 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:59.811329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:59.812112 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:23:59.812114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:23:59.812625 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:23:59.812626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:23:59.812686 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:59.813181 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:23:59.813187 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:23:59.815569 (+ 2382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:23:59.817541 (+ 1972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:23:59.817542 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:23:59.817543 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:23:59.817769 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:23:59.817982 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:23:59.817983 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:23:59.817983 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:23:59.818041 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:23:59.818324 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:23:59.818325 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:23:59.818369 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 20:23:59.818432 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 20:23:59.818534 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 20:23:59.819308 25576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:23:59.810034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46148 (local address 127.3.18.60:41701)
0928 20:23:59.810079 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:59.810081 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:59.810090 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:23:59.810158 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:59.810159 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:59.810170 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:59.810227 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:59.810515 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:59.811236 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:59.812210 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:23:59.812497 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:23:59.812558 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:23:59.815691 (+ 3133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:23:59.815693 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:23:59.815693 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:23:59.815701 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:23:59.817445 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:59.817850 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:59.817852 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:59.817852 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:59.817909 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:23:59.818106 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:23:59.818107 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:23:59.818107 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:23:59.818237 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:23:59.818296 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 20:23:59.819136 (+ 840us) server_negotiation.cc:299] Negotiation successful
0928 20:23:59.819217 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 20:23:59.822611 25497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d06a1f0a4eae436ab1219f5797104e14: Bootstrap starting.
I20260928 20:23:59.823421 25497 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d06a1f0a4eae436ab1219f5797104e14: Neither blocks nor log segments found. Creating new log.
I20260928 20:23:59.823736 25497 log.cc:824] T 00000000000000000000000000000000 P d06a1f0a4eae436ab1219f5797104e14: Log is configured to *not* fsync() on all Append() calls
I20260928 20:23:59.824667 25497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d06a1f0a4eae436ab1219f5797104e14: No bootstrap required, opened a new log
I20260928 20:23:59.826738 25497 raft_consensus.cc:348] T 00000000000000000000000000000000 P d06a1f0a4eae436ab1219f5797104e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3fde98bbb314312a92222889171dde1" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36647 } } peers { permanent_uuid: "d06a1f0a4eae436ab1219f5797104e14" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 44471 } } peers { permanent_uuid: "be945e9efd744a6eabb4f57b82602d7c" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 41701 } }
I20260928 20:23:59.826905 25497 raft_consensus.cc:729] T 00000000000000000000000000000000 P d06a1f0a4eae436ab1219f5797104e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d06a1f0a4eae436ab1219f5797104e14, State: Initialized, Role: FOLLOWER
I20260928 20:23:59.827004 25497 consensus_queue.cc:260] T 00000000000000000000000000000000 P d06a1f0a4eae436ab1219f5797104e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3fde98bbb314312a92222889171dde1" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36647 } } peers { permanent_uuid: "d06a1f0a4eae436ab1219f5797104e14" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 44471 } } peers { permanent_uuid: "be945e9efd744a6eabb4f57b82602d7c" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 41701 } }
I20260928 20:23:59.827350 25497 sys_catalog.cc:564] T 00000000000000000000000000000000 P d06a1f0a4eae436ab1219f5797104e14 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:23:59.827423 25470 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d06a1f0a4eae436ab1219f5797104e14 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:23:59.828495 25470 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d06a1f0a4eae436ab1219f5797104e14 [term 1 FOLLOWER]: Refusing update from remote peer d3fde98bbb314312a92222889171dde1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:23:59.829115 25592 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d3fde98bbb314312a92222889171dde1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d06a1f0a4eae436ab1219f5797104e14" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 44471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:23:59.829349 25622 sys_catalog.cc:455] T 00000000000000000000000000000000 P d06a1f0a4eae436ab1219f5797104e14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3fde98bbb314312a92222889171dde1" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36647 } } peers { permanent_uuid: "d06a1f0a4eae436ab1219f5797104e14" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 44471 } } peers { permanent_uuid: "be945e9efd744a6eabb4f57b82602d7c" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 41701 } } }
I20260928 20:23:59.829478 25622 sys_catalog.cc:458] T 00000000000000000000000000000000 P d06a1f0a4eae436ab1219f5797104e14 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 20:23:59.828718 25471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d3fde98bbb314312a92222889171dde1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d06a1f0a4eae436ab1219f5797104e14"
W20260928 20:23:59.829779 25633 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d06a1f0a4eae436ab1219f5797104e14: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 20:23:59.829835 25633 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 20:23:59.830586 25471 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d06a1f0a4eae436ab1219f5797104e14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3fde98bbb314312a92222889171dde1 in term 1.
I20260928 20:23:59.832106 25622 sys_catalog.cc:455] T 00000000000000000000000000000000 P d06a1f0a4eae436ab1219f5797104e14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3fde98bbb314312a92222889171dde1. Latest consensus state: current_term: 1 leader_uuid: "d3fde98bbb314312a92222889171dde1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3fde98bbb314312a92222889171dde1" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36647 } } peers { permanent_uuid: "d06a1f0a4eae436ab1219f5797104e14" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 44471 } } peers { permanent_uuid: "be945e9efd744a6eabb4f57b82602d7c" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 41701 } } }
I20260928 20:23:59.832276 25622 sys_catalog.cc:458] T 00000000000000000000000000000000 P d06a1f0a4eae436ab1219f5797104e14 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 20:23:59.833395 25622 sys_catalog.cc:455] T 00000000000000000000000000000000 P d06a1f0a4eae436ab1219f5797104e14 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d3fde98bbb314312a92222889171dde1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3fde98bbb314312a92222889171dde1" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36647 } } peers { permanent_uuid: "d06a1f0a4eae436ab1219f5797104e14" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 44471 } } peers { permanent_uuid: "be945e9efd744a6eabb4f57b82602d7c" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 41701 } } }
I20260928 20:23:59.833537 25622 sys_catalog.cc:458] T 00000000000000000000000000000000 P d06a1f0a4eae436ab1219f5797104e14 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 20:23:59.841730 25635 mvcc.cc:205] Tried to move back new op lower bound from 7334408355047870464 to 7334408354956144640. Current Snapshot: MvccSnapshot[applied={T|T < 7334408354967961600}]
W20260928 20:23:59.856572 25501 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:23:59.854480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44130 (local address 127.3.18.61:44471)
0928 20:23:59.854531 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:59.854534 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:59.855517 (+ 983us) server_negotiation.cc:407] Connection header received
0928 20:23:59.855655 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:59.855657 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:59.855672 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:59.855705 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:59.856490 (+ 785us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44130: BlockingRecv error: recv got EOF from 127.0.0.1:44130 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260928 20:23:59.857497 25576 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:23:59.854749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46158 (local address 127.3.18.60:41701)
0928 20:23:59.854794 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:59.854796 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:59.856871 (+ 2075us) server_negotiation.cc:407] Connection header received
0928 20:23:59.856979 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:59.856980 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:59.856992 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:59.857022 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:59.857429 (+ 407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46158: BlockingRecv error: recv got EOF from 127.0.0.1:46158 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260928 20:23:59.858302 25425 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:23:59.854956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39278 (local address 127.3.18.62:36647)
0928 20:23:59.854997 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:59.854999 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:59.857749 (+ 2750us) server_negotiation.cc:407] Connection header received
0928 20:23:59.857852 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:59.857853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:59.857864 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:59.857895 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:59.858236 (+ 341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39278: BlockingRecv error: recv got EOF from 127.0.0.1:39278 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 20:23:59.861378 25501 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:23:59.859995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44136 (local address 127.3.18.61:44471)
0928 20:23:59.860038 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:59.860040 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:59.860744 (+ 704us) server_negotiation.cc:407] Connection header received
0928 20:23:59.860886 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:59.860887 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:59.860911 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:59.860941 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:59.861324 (+ 383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44136: BlockingRecv error: recv got EOF from 127.0.0.1:44136 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 20:23:59.864794 25501 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:23:59.863869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44146 (local address 127.3.18.61:44471)
0928 20:23:59.863910 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:59.863912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:59.864205 (+ 293us) server_negotiation.cc:407] Connection header received
0928 20:23:59.864309 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:59.864310 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:59.864322 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:59.864353 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:59.864728 (+ 375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44146: BlockingRecv error: recv got EOF from 127.0.0.1:44146 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 20:23:59.869048 25501 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:23:59.868169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44162 (local address 127.3.18.61:44471)
0928 20:23:59.868211 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:59.868213 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:59.868456 (+ 243us) server_negotiation.cc:407] Connection header received
0928 20:23:59.868550 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:59.868551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:59.868561 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:59.868589 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:59.868989 (+ 400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44162: BlockingRecv error: recv got EOF from 127.0.0.1:44162 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 20:23:59.875281 25501 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:23:59.874404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44164 (local address 127.3.18.61:44471)
0928 20:23:59.874445 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:59.874447 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:59.874687 (+ 240us) server_negotiation.cc:407] Connection header received
0928 20:23:59.874787 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:59.874788 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:59.874799 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:59.874828 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:59.875217 (+ 389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44164: BlockingRecv error: recv got EOF from 127.0.0.1:44164 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 20:23:59.885562 25501 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:23:59.884669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44180 (local address 127.3.18.61:44471)
0928 20:23:59.884711 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:59.884713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:59.884964 (+ 251us) server_negotiation.cc:407] Connection header received
0928 20:23:59.885071 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:59.885072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:59.885084 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:59.885126 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:59.885503 (+ 377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44180: BlockingRecv error: recv got EOF from 127.0.0.1:44180 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 20:23:59.904098 25501 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:23:59.903022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44196 (local address 127.3.18.61:44471)
0928 20:23:59.903069 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:59.903071 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:59.903427 (+ 356us) server_negotiation.cc:407] Connection header received
0928 20:23:59.903569 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:59.903570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:59.903581 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:59.903612 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:59.904039 (+ 427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44196: BlockingRecv error: recv got EOF from 127.0.0.1:44196 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 20:23:59.938777 25501 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:23:59.937669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44212 (local address 127.3.18.61:44471)
0928 20:23:59.937718 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 20:23:59.937720 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:23:59.938091 (+ 371us) server_negotiation.cc:407] Connection header received
0928 20:23:59.938223 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:23:59.938224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:23:59.938236 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:23:59.938269 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:23:59.938707 (+ 438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44212: BlockingRecv error: recv got EOF from 127.0.0.1:44212 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260928 20:24:00.005409 25501 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:24:00.004519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44216 (local address 127.3.18.61:44471)
0928 20:24:00.004580 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:00.004583 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:00.004726 (+ 143us) server_negotiation.cc:407] Connection header received
0928 20:24:00.004803 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:00.004804 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:00.004814 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:00.004867 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:00.005337 (+ 470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44216: BlockingRecv error: recv got EOF from 127.0.0.1:44216 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260928 20:24:00.135424 25501 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:24:00.134737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44228 (local address 127.3.18.61:44471)
0928 20:24:00.134811 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:00.134813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:00.134824 (+ 11us) server_negotiation.cc:407] Connection header received
0928 20:24:00.134868 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:00.134868 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:00.134879 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:00.134947 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:00.135374 (+ 427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44228: BlockingRecv error: recv got EOF from 127.0.0.1:44228 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 20:24:00.217799 25592 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3fde98bbb314312a92222889171dde1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d3fde98bbb314312a92222889171dde1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3fde98bbb314312a92222889171dde1" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 36647 } } peers { permanent_uuid: "d06a1f0a4eae436ab1219f5797104e14" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 44471 } } peers { permanent_uuid: "be945e9efd744a6eabb4f57b82602d7c" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 41701 } } }
I20260928 20:24:00.218009 25592 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3fde98bbb314312a92222889171dde1 [sys.catalog]: This master's current role is: LEADER
W20260928 20:24:00.393424 25501 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:24:00.392836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44230 (local address 127.3.18.61:44471)
0928 20:24:00.392907 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:00.392909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:00.392917 (+ 8us) server_negotiation.cc:407] Connection header received
0928 20:24:00.392943 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:00.392944 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:00.392953 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:00.393018 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:00.393374 (+ 356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44230: BlockingRecv error: recv got EOF from 127.0.0.1:44230 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 20:24:00.774771 25589 catalog_manager.cc:1259] Loaded cluster ID: ae49c499b47844a08fe47b2a681181e4
I20260928 20:24:00.774852 25589 catalog_manager.cc:1552] T 00000000000000000000000000000000 P be945e9efd744a6eabb4f57b82602d7c: loading cluster ID for follower catalog manager: success
I20260928 20:24:00.775730 25589 catalog_manager.cc:1574] T 00000000000000000000000000000000 P be945e9efd744a6eabb4f57b82602d7c: acquiring CA information for follower catalog manager: success
I20260928 20:24:00.776103 25589 catalog_manager.cc:1602] T 00000000000000000000000000000000 P be945e9efd744a6eabb4f57b82602d7c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 20:24:00.830289 25633 catalog_manager.cc:1259] Loaded cluster ID: ae49c499b47844a08fe47b2a681181e4
I20260928 20:24:00.830379 25633 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d06a1f0a4eae436ab1219f5797104e14: loading cluster ID for follower catalog manager: success
I20260928 20:24:00.831226 25633 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d06a1f0a4eae436ab1219f5797104e14: acquiring CA information for follower catalog manager: success
I20260928 20:24:00.831535 25633 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d06a1f0a4eae436ab1219f5797104e14: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 20:24:00.909632 25705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:00.906702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44232 (local address 127.3.18.61:44471)
0928 20:24:00.906823 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:00.906825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:00.906874 (+ 49us) server_negotiation.cc:407] Connection header received
0928 20:24:00.906974 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:00.906974 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:00.907030 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:00.907114 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:24:00.907302 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:00.907905 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:00.908780 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:00.908987 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:00.909043 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:00.909228 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:24:00.909343 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:24:00.909407 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:00.909444 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 20:24:00.909462 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":70,"threads_started":1}
I20260928 20:24:00.914184 25711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:00.911706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46166 (local address 127.3.18.60:41701)
0928 20:24:00.911847 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:00.911850 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:00.911864 (+ 14us) server_negotiation.cc:407] Connection header received
0928 20:24:00.911910 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:00.911911 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:00.911946 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:00.912022 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:24:00.912157 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:00.912705 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:00.913470 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:00.913695 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:00.913749 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:00.913873 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:24:00.913974 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:24:00.914034 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:00.914068 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 20:24:00.914085 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":78,"threads_started":1}
I20260928 20:24:00.919662 25717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:00.916153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39288 (local address 127.3.18.62:36647)
0928 20:24:00.916283 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:00.916286 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:00.916560 (+ 274us) server_negotiation.cc:407] Connection header received
0928 20:24:00.916694 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:00.916695 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:00.916751 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:00.916838 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:24:00.917009 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:00.917883 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:00.918739 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:00.918977 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:00.919066 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:00.919274 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:24:00.919367 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:24:00.919445 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:00.919506 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 20:24:00.919527 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":72,"threads_started":1}
I20260928 20:24:00.919750 25705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:00.916099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44248 (local address 127.3.18.61:44471)
0928 20:24:00.916196 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:00.916199 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:00.916401 (+ 202us) server_negotiation.cc:407] Connection header received
0928 20:24:00.916479 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:00.916480 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:00.916494 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:00.916595 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:24:00.916734 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:00.917557 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:00.918753 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:00.919090 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:00.919351 (+ 261us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:00.919381 (+ 30us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:24:00.919415 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:24:00.919611 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:00.919628 (+ 17us) server_negotiation.cc:299] Negotiation successful
0928 20:24:00.919653 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 20:24:00.920845 25711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:00.916171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46172 (local address 127.3.18.60:41701)
0928 20:24:00.916229 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:00.916231 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:00.916545 (+ 314us) server_negotiation.cc:407] Connection header received
0928 20:24:00.916655 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:00.916656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:00.916668 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:00.916732 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:24:00.917826 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:00.918606 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:00.919801 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:00.920099 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:00.920174 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:00.920566 (+ 392us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:24:00.920605 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:24:00.920684 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:00.920726 (+ 42us) server_negotiation.cc:299] Negotiation successful
0928 20:24:00.920753 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 20:24:00.921759 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 25354
I20260928 20:24:00.927455 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 25429
I20260928 20:24:00.933108 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 25504
2026-09-28T20:24:00Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2839 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:24:01 dist-test-slave-b83v krb5kdc[25722](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:24:01 dist-test-slave-b83v krb5kdc[25722](info): set up 2 sockets
Sep 28 20:24:01 dist-test-slave-b83v krb5kdc[25722](info): commencing operation
krb5kdc: starting...
W20260928 20:24:02.002161 3144 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/krb5kdc/test-user.keytab.
Sep 28 20:24:02 dist-test-slave-b83v krb5kdc[25722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627042, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:24:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:24:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.18.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab.
Entry for principal kudu/127.3.18.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab.
I20260928 20:24:02.139604 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:37885
--webserver_interface=127.3.18.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:41963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.18.62:37885,127.3.18.61:32869,127.3.18.60:32889
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:24:02.225112 25735 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 20:24:02.225419 25735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:24:02.225455 25735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:24:02.225507 25735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:24:02.226788 25735 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:24:02.226874 25735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:24:02.226897 25735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:24:02.226918 25735 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:24:02.226945 25735 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:24:02.229008 25735 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:41963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.18.62:37885,127.3.18.61:32869,127.3.18.60:32889
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.62.keytab
--principal=kudu/127.3.18.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.62:37885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.18.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:24:02.229368 25735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:24:02.229637 25735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:24:02.235241 25741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:02.235203 25743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:02.235350 25735 server_base.cc:1034] running on GCE node
W20260928 20:24:02.235292 25740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:02.235864 25735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:24:02.236181 25735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:24:02.237351 25735 hybrid_clock.cc:648] HybridClock initialized: now 1790627042237339 us; error 32 us; skew 500 ppm
Sep 28 20:24:02 dist-test-slave-b83v krb5kdc[25722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627042, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:24:02.240039 25735 init.cc:377] Logged in from keytab as kudu/127.3.18.62@KRBTEST.COM (short username kudu)
I20260928 20:24:02.240449 25735 webserver.cc:460] Webserver started at http://127.3.18.62:45883/ using document root <none> and password file <none>
I20260928 20:24:02.240715 25735 fs_manager.cc:359] Metadata directory not provided
I20260928 20:24:02.240761 25735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:24:02.240916 25735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:24:02.242103 25735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/minicluster-data/master-0/data/instance:
uuid: "173110791c984c52bb03c75cffa3bb9d"
format_stamp: "Formatted at 2026-09-28 20:24:02 on dist-test-slave-b83v"
server_key: "70ebbf26bcf0dbef883f658242211615"
server_key_iv: "515206b4e542243376038d001413c4e9"
server_key_version: "encryptionkey@0"
I20260928 20:24:02.242472 25735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/minicluster-data/master-0/wal/instance:
uuid: "173110791c984c52bb03c75cffa3bb9d"
format_stamp: "Formatted at 2026-09-28 20:24:02 on dist-test-slave-b83v"
server_key: "70ebbf26bcf0dbef883f658242211615"
server_key_iv: "515206b4e542243376038d001413c4e9"
server_key_version: "encryptionkey@0"
I20260928 20:24:02.243965 25735 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260928 20:24:02.244793 25750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:02.244976 25735 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:24:02.245072 25735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/minicluster-data/master-0/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/minicluster-data/master-0/wal
uuid: "173110791c984c52bb03c75cffa3bb9d"
format_stamp: "Formatted at 2026-09-28 20:24:02 on dist-test-slave-b83v"
server_key: "70ebbf26bcf0dbef883f658242211615"
server_key_iv: "515206b4e542243376038d001413c4e9"
server_key_version: "encryptionkey@0"
I20260928 20:24:02.245151 25735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:24:02.259557 25735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:24:02.261965 25735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:24:02.262180 25735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:24:02.266891 25735 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.62:37885
I20260928 20:24:02.266944 25802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.62:37885 every 8 connection(s)
I20260928 20:24:02.267362 25735 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/minicluster-data/master-0/data/info.pb
I20260928 20:24:02.268481 25803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:02.270164 25803 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.18.62" port: 37885 } has no permanent_uuid. Determining permanent_uuid...
I20260928 20:24:02.275632 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 25735
I20260928 20:24:02.275730 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/minicluster-data/master-0/wal/instance
I20260928 20:24:02.275995 3144 external_mini_cluster.cc:1428] Setting key 5ac1950c96daf1c5a2154fa8680b3c3f
Sep 28 20:24:02 dist-test-slave-b83v krb5kdc[25722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627042, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.62@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:24:02.281962 25805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:02.270931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:37885 (local address 127.0.0.1:51454)
0928 20:24:02.271172 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:02.271215 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:02.271699 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:02.272135 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:02.272138 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:02.272431 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:02.272613 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:02.272618 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:02.273752 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:02.273753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:02.274612 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:02.274615 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:02.275200 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:02.275201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:02.275268 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:02.276016 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:02.276046 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:02.277612 (+ 1566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:02.280247 (+ 2635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:02.280254 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:02.280256 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:02.280482 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:02.280793 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:02.280794 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:02.280794 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:02.280884 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:02.281429 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:02.281436 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:02.281516 (+ 80us) client_negotiation.cc:769] Sending connection context
0928 20:24:02.281614 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 20:24:02.281764 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":81,"threads_started":1}
I20260928 20:24:02.281984 25806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:02.271114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51454 (local address 127.3.18.62:37885)
0928 20:24:02.271248 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:02.271252 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:02.271286 (+ 34us) server_negotiation.cc:407] Connection header received
0928 20:24:02.271934 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:02.271937 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:02.271955 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:02.272049 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:02.272739 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:02.273644 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:02.274714 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:02.275068 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:02.275169 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:02.277713 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:02.277722 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:02.277723 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:02.277732 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:02.280131 (+ 2399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:02.280576 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:02.280581 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:02.280582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:02.280702 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:02.280958 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:02.280959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:02.280960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:02.281325 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:02.281430 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:02.281672 (+ 242us) server_negotiation.cc:299] Negotiation successful
0928 20:24:02.281861 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":101,"thread_start_us":69,"threads_started":1}
I20260928 20:24:02.282850 25803 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.18.61" port: 32869 } has no permanent_uuid. Determining permanent_uuid...
I20260928 20:24:02.283258 25805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:02.283136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.61:32869 (local address 127.0.0.1:49668)
0928 20:24:02.283184 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:02.283236 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.61:32869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 20:24:02.283387 25751 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.18.61:32869: connect: Connection refused (error 111)
I20260928 20:24:02.283895 25805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:02.283735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.61:32869 (local address 127.0.0.1:49676)
0928 20:24:02.283822 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:02.283858 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.61:32869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260928 20:24:02.284017 25803 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.18.61:32869: Network error: Client connection negotiation failed: client connection to 127.3.18.61:32869: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.3.18.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.61.keytab.
Entry for principal kudu/127.3.18.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.61.keytab.
I20260928 20:24:02.304028 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.61:32869
--webserver_interface=127.3.18.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.61.keytab
--principal=kudu/127.3.18.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:41963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.18.62:37885,127.3.18.61:32869,127.3.18.60:32889
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 20:24:02.333294 25803 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.18.61" port: 32869 } attempt: 1
I20260928 20:24:02.333890 25805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:02.333737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.61:32869 (local address 127.0.0.1:49678)
0928 20:24:02.333802 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:02.333853 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.61:32869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 20:24:02.334298 25805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:02.334194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.61:32869 (local address 127.0.0.1:49688)
0928 20:24:02.334244 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:02.334277 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.61:32869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 20:24:02.334399 25803 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.18.61:32869: Network error: Client connection negotiation failed: client connection to 127.3.18.61:32869: connect: Connection refused (error 111)
W20260928 20:24:02.391950 25810 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 20:24:02.392189 25810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:24:02.392207 25810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:24:02.392228 25810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:24:02.393420 25810 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:24:02.393468 25810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:24:02.393483 25810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:24:02.393496 25810 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:24:02.393507 25810 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:24:02.395067 25810 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:41963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.18.62:37885,127.3.18.61:32869,127.3.18.60:32889
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.61.keytab
--principal=kudu/127.3.18.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.61:32869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.18.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:24:02.395349 25810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:24:02.395601 25810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:24:02.401233 25818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:02.401353 25810 server_base.cc:1034] running on GCE node
W20260928 20:24:02.401238 25816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:02.401238 25815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:02.401871 25810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:24:02.402206 25810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:24:02.402522 25803 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.18.61" port: 32869 } attempt: 2
I20260928 20:24:02.402880 25805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:02.402767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.61:32869 (local address 127.0.0.1:49704)
0928 20:24:02.402820 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:02.402853 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.61:32869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 20:24:02.403262 25805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:02.403175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.61:32869 (local address 127.0.0.1:49714)
0928 20:24:02.403218 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:02.403240 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.61:32869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 20:24:02.403366 25803 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.18.61:32869: Network error: Client connection negotiation failed: client connection to 127.3.18.61:32869: connect: Connection refused (error 111)
I20260928 20:24:02.403441 25810 hybrid_clock.cc:648] HybridClock initialized: now 1790627042403424 us; error 43 us; skew 500 ppm
Sep 28 20:24:02 dist-test-slave-b83v krb5kdc[25722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627042, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:24:02.406179 25810 init.cc:377] Logged in from keytab as kudu/127.3.18.61@KRBTEST.COM (short username kudu)
I20260928 20:24:02.406688 25810 webserver.cc:460] Webserver started at http://127.3.18.61:41227/ using document root <none> and password file <none>
I20260928 20:24:02.406932 25810 fs_manager.cc:359] Metadata directory not provided
I20260928 20:24:02.406998 25810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:24:02.407166 25810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:24:02.408553 25810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/minicluster-data/master-1/data/instance:
uuid: "1b69b1414b084a938036884696a8737c"
format_stamp: "Formatted at 2026-09-28 20:24:02 on dist-test-slave-b83v"
server_key: "1a8c45d50a7fdfc6e64830de27b66514"
server_key_iv: "3e30fad0dd46e8fc70c7aba60e973510"
server_key_version: "encryptionkey@0"
I20260928 20:24:02.409010 25810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/minicluster-data/master-1/wal/instance:
uuid: "1b69b1414b084a938036884696a8737c"
format_stamp: "Formatted at 2026-09-28 20:24:02 on dist-test-slave-b83v"
server_key: "1a8c45d50a7fdfc6e64830de27b66514"
server_key_iv: "3e30fad0dd46e8fc70c7aba60e973510"
server_key_version: "encryptionkey@0"
I20260928 20:24:02.410511 25810 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 20:24:02.411391 25825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:02.411623 25810 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:24:02.411752 25810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/minicluster-data/master-1/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/minicluster-data/master-1/wal
uuid: "1b69b1414b084a938036884696a8737c"
format_stamp: "Formatted at 2026-09-28 20:24:02 on dist-test-slave-b83v"
server_key: "1a8c45d50a7fdfc6e64830de27b66514"
server_key_iv: "3e30fad0dd46e8fc70c7aba60e973510"
server_key_version: "encryptionkey@0"
I20260928 20:24:02.411888 25810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:24:02.426222 25810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:24:02.429069 25810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:24:02.429273 25810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:24:02.434192 25810 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.61:32869
I20260928 20:24:02.434227 25877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.61:32869 every 8 connection(s)
I20260928 20:24:02.434641 25810 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/minicluster-data/master-1/data/info.pb
I20260928 20:24:02.435717 25878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:02.437453 25878 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.18.62" port: 37885 } has no permanent_uuid. Determining permanent_uuid...
I20260928 20:24:02.439881 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 25810
I20260928 20:24:02.439988 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/minicluster-data/master-1/wal/instance
I20260928 20:24:02.440227 3144 external_mini_cluster.cc:1428] Setting key 30a66fff2055f5eccc621af40d9c4f3e
Sep 28 20:24:02 dist-test-slave-b83v krb5kdc[25722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627042, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.61@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.18.60@KRBTEST.COM; defaulting to no policy
I20260928 20:24:02.447708 25806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:02.438209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51464 (local address 127.3.18.62:37885)
0928 20:24:02.438280 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:02.438283 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:02.438543 (+ 260us) server_negotiation.cc:407] Connection header received
0928 20:24:02.439005 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:02.439006 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:02.439018 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:02.439070 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:02.439747 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:02.440518 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:02.441590 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:02.441825 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:02.441912 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:02.444529 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:02.444531 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:02.444531 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:02.444538 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:02.446116 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:02.446700 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:02.446702 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:02.446702 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:02.446762 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:02.447011 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:02.447012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:02.447013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:02.447138 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:02.447226 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:02.447513 (+ 287us) server_negotiation.cc:299] Negotiation successful
0928 20:24:02.447614 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 20:24:02.447939 25880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:02.438215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:37885 (local address 127.0.0.1:51464)
0928 20:24:02.438417 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:02.438456 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:02.438923 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:02.439126 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:02.439129 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:02.439408 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:02.439654 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:02.439659 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:02.440614 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:02.440615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:02.441488 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:02.441491 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:02.441937 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:02.441938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:02.442074 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:02.442672 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:02.442680 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:02.444364 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:02.446250 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:02.446259 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:02.446262 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:02.446602 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:02.446827 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:02.446828 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:02.446828 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:02.446948 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:02.447220 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:02.447228 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:02.447384 (+ 156us) client_negotiation.cc:769] Sending connection context
0928 20:24:02.447492 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 20:24:02.447661 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":97,"threads_started":1}
I20260928 20:24:02.448752 25878 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.18.61" port: 32869 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.18.60@KRBTEST.COM" created.
Sep 28 20:24:02 dist-test-slave-b83v krb5kdc[25722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627042, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.61@KRBTEST.COM for kudu/127.3.18.61@KRBTEST.COM
I20260928 20:24:02.457597 25880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:02.449047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.61:32869 (local address 127.0.0.1:49726)
0928 20:24:02.449290 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:02.449299 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:02.449337 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:02.449464 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:02.449464 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:02.449583 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:02.449642 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:02.449644 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:02.450765 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:02.450767 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:02.451397 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:02.451400 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:02.452227 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:02.452228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:02.452305 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:02.452835 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:02.452840 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:02.454390 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:02.456463 (+ 2073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:02.456464 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:02.456465 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:02.456641 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:02.456867 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:02.456868 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:02.456868 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:02.456923 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:02.457231 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:02.457231 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:02.457270 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 20:24:02.457336 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 20:24:02.457466 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":2}
I20260928 20:24:02.457664 25882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:02.449148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49726 (local address 127.3.18.61:32869)
0928 20:24:02.449312 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:02.449316 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:02.449337 (+ 21us) server_negotiation.cc:407] Connection header received
0928 20:24:02.449375 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:02.449377 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:02.449389 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:02.449452 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:02.449730 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:02.450695 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:02.451500 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:02.452115 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:02.452203 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:02.454495 (+ 2292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:02.454506 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:02.454507 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:02.454514 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:02.456347 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:02.456737 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:02.456741 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:02.456742 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:02.456803 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:02.456989 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:02.456990 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:02.456991 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:02.457164 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:02.457257 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:02.457366 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 20:24:02.457532 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":92,"server-negotiator.queue_time_us":129,"thread_start_us":99,"threads_started":1}
I20260928 20:24:02.458287 25878 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.18.60" port: 32889 } has no permanent_uuid. Determining permanent_uuid...
I20260928 20:24:02.458685 25880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:02.458553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.60:32889 (local address 127.0.0.1:35898)
0928 20:24:02.458607 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:02.458655 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.60:32889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 20:24:02.458849 25826 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.18.60:32889: connect: Connection refused (error 111)
I20260928 20:24:02.459292 25880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:02.459202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.60:32889 (local address 127.0.0.1:35904)
0928 20:24:02.459243 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:02.459267 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.60:32889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 20:24:02.459398 25878 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.18.60:32889: Network error: Client connection negotiation failed: client connection to 127.3.18.60:32889: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.18.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.60.keytab.
Entry for principal kudu/127.3.18.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.60.keytab.
I20260928 20:24:02.471920 3144 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2yfqO7/build/release/bin/kudu
/tmp/dist-test-task2yfqO7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.60:32889
--webserver_interface=127.3.18.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.60.keytab
--principal=kudu/127.3.18.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.18.20:41963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.18.62:37885,127.3.18.61:32869,127.3.18.60:32889
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 20:24:02.494535 25803 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.18.61" port: 32869 } attempt: 3
Sep 28 20:24:02 dist-test-slave-b83v krb5kdc[25722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627042, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.62@KRBTEST.COM for kudu/127.3.18.61@KRBTEST.COM
I20260928 20:24:02.502386 25805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:02.494972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.61:32869 (local address 127.0.0.1:49740)
0928 20:24:02.495045 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:02.495060 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:02.495147 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:02.495310 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:02.495311 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:02.495504 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:02.495575 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:02.495577 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:02.496437 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:02.496439 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:02.497020 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:02.497021 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:02.497456 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:02.497457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:02.497564 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:02.498092 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:02.498097 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:02.499428 (+ 1331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:02.501232 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:02.501233 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:02.501234 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:02.501451 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:02.501682 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:02.501683 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:02.501683 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:02.501747 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:02.502013 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:02.502014 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:02.502074 (+ 60us) client_negotiation.cc:769] Sending connection context
0928 20:24:02.502161 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 20:24:02.502275 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 20:24:02.502386 25882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:02.495072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49740 (local address 127.3.18.61:32869)
0928 20:24:02.495160 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:02.495163 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:02.495175 (+ 12us) server_negotiation.cc:407] Connection header received
0928 20:24:02.495219 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:02.495220 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:02.495238 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:02.495304 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:02.495676 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:02.496325 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:02.497129 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:02.497373 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:02.497458 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:02.499521 (+ 2063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:02.499523 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:02.499523 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:02.499532 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:02.501125 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:02.501547 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:02.501548 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:02.501549 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:02.501611 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:02.501816 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:02.501818 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:02.501819 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:02.501945 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:02.502026 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:02.502186 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 20:24:02.502277 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 20:24:02.502861 25803 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.18.60" port: 32889 } has no permanent_uuid. Determining permanent_uuid...
I20260928 20:24:02.503247 25805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:02.503132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.60:32889 (local address 127.0.0.1:35914)
0928 20:24:02.503183 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:02.503215 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.60:32889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 20:24:02.503618 25805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:02.503509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.60:32889 (local address 127.0.0.1:35924)
0928 20:24:02.503563 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:02.503588 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.60:32889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 20:24:02.503736 25803 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.18.60:32889: Network error: Client connection negotiation failed: client connection to 127.3.18.60:32889: connect: Connection refused (error 111)
I20260928 20:24:02.508646 25878 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.18.60" port: 32889 } attempt: 1
I20260928 20:24:02.509186 25880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:02.509013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.60:32889 (local address 127.0.0.1:35940)
0928 20:24:02.509075 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:02.509155 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.60:32889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 20:24:02.509596 25880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:02.509509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.60:32889 (local address 127.0.0.1:35950)
0928 20:24:02.509554 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:02.509580 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.60:32889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 20:24:02.509696 25878 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.18.60:32889: Network error: Client connection negotiation failed: client connection to 127.3.18.60:32889: connect: Connection refused (error 111)
I20260928 20:24:02.534933 25803 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.18.60" port: 32889 } attempt: 1
I20260928 20:24:02.535524 25805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:02.535330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.60:32889 (local address 127.0.0.1:35956)
0928 20:24:02.535417 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:02.535478 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.60:32889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 20:24:02.535981 25805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:02.535869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.60:32889 (local address 127.0.0.1:35972)
0928 20:24:02.535919 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:02.535956 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.60:32889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 20:24:02.536196 25803 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.18.60:32889: Network error: Client connection negotiation failed: client connection to 127.3.18.60:32889: connect: Connection refused (error 111)
W20260928 20:24:02.557763 25885 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 20:24:02.557986 25885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:24:02.558002 25885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:24:02.558023 25885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:24:02.559132 25885 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:24:02.559170 25885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:24:02.559185 25885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:24:02.559196 25885 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:24:02.559209 25885 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:24:02.561056 25885 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.18.20:41963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.18.62:37885,127.3.18.61:32869,127.3.18.60:32889
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/krb5kdc/kudu_127.3.18.60.keytab
--principal=kudu/127.3.18.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.18.60:32889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.18.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:24:02.561296 25885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:24:02.561503 25885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:24:02.566480 25891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:02.566550 25893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:02.566565 25890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:02.566809 25885 server_base.cc:1034] running on GCE node
I20260928 20:24:02.567116 25885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:24:02.567457 25885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:24:02.568616 25885 hybrid_clock.cc:648] HybridClock initialized: now 1790627042568581 us; error 48 us; skew 500 ppm
Sep 28 20:24:02 dist-test-slave-b83v krb5kdc[25722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627042, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:24:02.571213 25885 init.cc:377] Logged in from keytab as kudu/127.3.18.60@KRBTEST.COM (short username kudu)
I20260928 20:24:02.571645 25885 webserver.cc:460] Webserver started at http://127.3.18.60:38167/ using document root <none> and password file <none>
I20260928 20:24:02.571915 25885 fs_manager.cc:359] Metadata directory not provided
I20260928 20:24:02.571965 25885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:24:02.572058 25885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:24:02.573167 25885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/minicluster-data/master-2/data/instance:
uuid: "4e120c52c2f5487c8ca16dfce5ce5198"
format_stamp: "Formatted at 2026-09-28 20:24:02 on dist-test-slave-b83v"
server_key: "b5a451dcf732e2f819beeb35baca4a7d"
server_key_iv: "ea1fb77a020d4d2a37b69bb45aab8a5b"
server_key_version: "encryptionkey@0"
I20260928 20:24:02.573493 25885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/minicluster-data/master-2/wal/instance:
uuid: "4e120c52c2f5487c8ca16dfce5ce5198"
format_stamp: "Formatted at 2026-09-28 20:24:02 on dist-test-slave-b83v"
server_key: "b5a451dcf732e2f819beeb35baca4a7d"
server_key_iv: "ea1fb77a020d4d2a37b69bb45aab8a5b"
server_key_version: "encryptionkey@0"
I20260928 20:24:02.574875 25885 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:24:02.575718 25900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:02.575981 25885 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:24:02.576086 25885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/minicluster-data/master-2/data,/tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/minicluster-data/master-2/wal
uuid: "4e120c52c2f5487c8ca16dfce5ce5198"
format_stamp: "Formatted at 2026-09-28 20:24:02 on dist-test-slave-b83v"
server_key: "b5a451dcf732e2f819beeb35baca4a7d"
server_key_iv: "ea1fb77a020d4d2a37b69bb45aab8a5b"
server_key_version: "encryptionkey@0"
I20260928 20:24:02.576185 25885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:24:02.577836 25878 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.18.60" port: 32889 } attempt: 2
I20260928 20:24:02.578238 25880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:02.578120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.60:32889 (local address 127.0.0.1:35988)
0928 20:24:02.578174 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:02.578210 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.60:32889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 20:24:02.578540 25880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:02.578456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.60:32889 (local address 127.0.0.1:36004)
0928 20:24:02.578496 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:02.578519 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.18.60:32889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 20:24:02.578681 25878 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.18.60:32889: Network error: Client connection negotiation failed: client connection to 127.3.18.60:32889: connect: Connection refused (error 111)
I20260928 20:24:02.600049 25885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:24:02.602542 25885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:24:02.602677 25885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:24:02.606869 25885 rpc_server.cc:307] RPC server started. Bound to: 127.3.18.60:32889
I20260928 20:24:02.607023 25952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.18.60:32889 every 8 connection(s)
I20260928 20:24:02.607283 25885 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/minicluster-data/master-2/data/info.pb
I20260928 20:24:02.607538 3144 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2yfqO7/build/release/bin/kudu as pid 25885
I20260928 20:24:02.607656 3144 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2yfqO7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790626764250411-3144-0/minicluster-data/master-2/wal/instance
I20260928 20:24:02.607935 3144 external_mini_cluster.cc:1428] Setting key 9f8e7bf6dd18c8d23394c11f90e06057
I20260928 20:24:02.608556 25953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:02.610507 25953 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.18.62" port: 37885 } has no permanent_uuid. Determining permanent_uuid...
I20260928 20:24:02.611338 25803 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.18.60" port: 32889 } attempt: 2
Sep 28 20:24:02 dist-test-slave-b83v krb5kdc[25722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627042, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.18.60@KRBTEST.COM
Sep 28 20:24:02 dist-test-slave-b83v krb5kdc[25722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627042, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.62@KRBTEST.COM for kudu/127.3.18.60@KRBTEST.COM
Sep 28 20:24:02 dist-test-slave-b83v krb5kdc[25722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627042, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.60@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:24:02.619436 25805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:02.611563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.60:32889 (local address 127.0.0.1:36020)
0928 20:24:02.611622 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:02.611632 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:02.611666 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:02.611925 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:02.611926 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:02.612089 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:02.612139 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:02.612141 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:02.613007 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:02.613008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:02.613499 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:02.613500 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:02.613890 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:02.613891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:02.613949 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:02.614663 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:02.614668 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:02.615967 (+ 1299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:02.618125 (+ 2158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:02.618126 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:02.618126 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:02.618312 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:02.618574 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:02.618575 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:02.618575 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:02.618639 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:02.619027 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:02.619028 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:02.619079 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 20:24:02.619226 (+ 147us) client_negotiation.cc:240] Negotiation successful
0928 20:24:02.619324 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 20:24:02.620069 25806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:02.611165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51472 (local address 127.3.18.62:37885)
0928 20:24:02.611231 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:02.611233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:02.611406 (+ 173us) server_negotiation.cc:407] Connection header received
0928 20:24:02.611464 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:02.611465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:02.611476 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:02.611525 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:02.612124 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:02.612876 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:02.614321 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:02.614508 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:02.614599 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:02.616783 (+ 2184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:02.616784 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:02.616784 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:02.616791 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:02.618670 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:02.619196 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:02.619197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:02.619197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:02.619248 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:02.619461 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:02.619462 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:02.619462 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:02.619608 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:02.619653 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:02.619895 (+ 242us) server_negotiation.cc:299] Negotiation successful
0928 20:24:02.619974 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 20:24:02.620196 25957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:02.611069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.62:37885 (local address 127.0.0.1:51472)
0928 20:24:02.611256 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:02.611331 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:02.611404 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:02.611573 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:02.611576 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:02.611987 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:02.612047 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:02.612049 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:02.612949 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:02.612950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:02.614244 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:02.614247 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:02.614623 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:02.614625 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:02.614711 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:02.615258 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:02.615268 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:02.616708 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:02.618782 (+ 2074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:02.618789 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:02.618791 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:02.619113 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:02.619323 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:02.619324 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:02.619324 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:02.619387 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:02.619701 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:02.619707 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:02.619755 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 20:24:02.619864 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 20:24:02.619970 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":123,"threads_started":1}
I20260928 20:24:02.620503 25956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:02.608634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36016 (local address 127.3.18.60:32889)
0928 20:24:02.608865 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:02.608876 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:02.608909 (+ 33us) server_negotiation.cc:407] Connection header received
0928 20:24:02.609335 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:02.609342 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:02.609388 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:02.609540 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:02.610197 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:02.611211 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:02.612278 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:02.612692 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:02.612820 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:02.616169 (+ 3349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:02.616170 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:02.616170 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:02.616177 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:02.618995 (+ 2818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:02.619415 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:02.619416 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:02.619416 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:02.619465 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:02.619659 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:02.619659 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:02.619660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:02.619792 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:02.619918 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:02.620322 (+ 404us) server_negotiation.cc:299] Negotiation successful
0928 20:24:02.620400 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":91,"threads_started":1}
I20260928 20:24:02.620535 25958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:02.611632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36020 (local address 127.3.18.60:32889)
0928 20:24:02.611796 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:02.611798 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:02.611806 (+ 8us) server_negotiation.cc:407] Connection header received
0928 20:24:02.611822 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:02.611823 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:02.611831 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:02.611898 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:02.612223 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:02.612940 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:02.613576 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:02.613810 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:02.613888 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:02.616056 (+ 2168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:02.616063 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:02.616063 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:02.616070 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:02.618037 (+ 1967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:02.618389 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:02.618393 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:02.618393 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:02.618503 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:02.618696 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:02.618697 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:02.618697 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:02.618949 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:02.619063 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:02.619294 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 20:24:02.619422 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":92,"threads_started":1}
I20260928 20:24:02.620743 25953 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.18.61" port: 32869 } has no permanent_uuid. Determining permanent_uuid...
I20260928 20:24:02.622363 25803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 173110791c984c52bb03c75cffa3bb9d: Bootstrap starting.
I20260928 20:24:02.622938 25803 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 173110791c984c52bb03c75cffa3bb9d: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:02.623179 25803 log.cc:824] T 00000000000000000000000000000000 P 173110791c984c52bb03c75cffa3bb9d: Log is configured to *not* fsync() on all Append() calls
I20260928 20:24:02.624073 25803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 173110791c984c52bb03c75cffa3bb9d: No bootstrap required, opened a new log
I20260928 20:24:02.626143 25803 raft_consensus.cc:348] T 00000000000000000000000000000000 P 173110791c984c52bb03c75cffa3bb9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "173110791c984c52bb03c75cffa3bb9d" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 37885 } } peers { permanent_uuid: "1b69b1414b084a938036884696a8737c" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 32869 } } peers { permanent_uuid: "4e120c52c2f5487c8ca16dfce5ce5198" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 32889 } }
I20260928 20:24:02.626295 25803 raft_consensus.cc:729] T 00000000000000000000000000000000 P 173110791c984c52bb03c75cffa3bb9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 173110791c984c52bb03c75cffa3bb9d, State: Initialized, Role: FOLLOWER
I20260928 20:24:02.626375 25803 consensus_queue.cc:260] T 00000000000000000000000000000000 P 173110791c984c52bb03c75cffa3bb9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "173110791c984c52bb03c75cffa3bb9d" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 37885 } } peers { permanent_uuid: "1b69b1414b084a938036884696a8737c" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 32869 } } peers { permanent_uuid: "4e120c52c2f5487c8ca16dfce5ce5198" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 32889 } }
I20260928 20:24:02.626672 25960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 173110791c984c52bb03c75cffa3bb9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "173110791c984c52bb03c75cffa3bb9d" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 37885 } } peers { permanent_uuid: "1b69b1414b084a938036884696a8737c" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 32869 } } peers { permanent_uuid: "4e120c52c2f5487c8ca16dfce5ce5198" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 32889 } } }
I20260928 20:24:02.626729 25803 sys_catalog.cc:564] T 00000000000000000000000000000000 P 173110791c984c52bb03c75cffa3bb9d [sys.catalog]: configured and running, proceeding with master startup.
Sep 28 20:24:02 dist-test-slave-b83v krb5kdc[25722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627042, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.60@KRBTEST.COM for kudu/127.3.18.61@KRBTEST.COM
W20260928 20:24:02.628782 25971 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 173110791c984c52bb03c75cffa3bb9d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 20:24:02.628834 25971 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 20:24:02.631160 25957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:02.621009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.61:32869 (local address 127.0.0.1:49746)
0928 20:24:02.621095 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:02.621105 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:02.621137 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:02.623564 (+ 2427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:02.623565 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:02.623751 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:02.623838 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:02.623840 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:02.624496 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:02.624497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:02.625271 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:02.625273 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:02.625624 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:02.625626 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:02.625694 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:02.626104 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:02.626109 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:02.627675 (+ 1566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:02.629955 (+ 2280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:02.629956 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:02.629956 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:02.630173 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:02.630397 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:02.630397 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:02.630398 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:02.630463 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:02.630826 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:02.630827 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:02.630875 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 20:24:02.630945 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 20:24:02.631048 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
Sep 28 20:24:02 dist-test-slave-b83v krb5kdc[25722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627042, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.18.62@KRBTEST.COM
I20260928 20:24:02.631752 25882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:02.621074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49746 (local address 127.3.18.61:32869)
0928 20:24:02.623405 (+ 2331us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:02.623407 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:02.623416 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:24:02.623450 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:02.623451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:02.623459 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:02.623559 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:02.623913 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:02.624419 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:02.625365 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:02.625550 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:02.625600 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:02.627792 (+ 2192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:02.627793 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:02.627793 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:02.627801 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:02.629861 (+ 2060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:02.630259 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:02.630260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:02.630260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:02.630320 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:02.630534 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:02.630535 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:02.630535 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:02.630736 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:02.630798 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:02.631563 (+ 765us) server_negotiation.cc:299] Negotiation successful
0928 20:24:02.631663 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2304}
I20260928 20:24:02.629515 25735 master_runner.cc:186] Error getting master registration for 127.3.18.61:32869: OK, instance_id { permanent_uuid: "1b69b1414b084a938036884696a8737c" instance_seqno: 1790627042432840 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 20:24:02.632447 25735 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 20:24:02.626811 25960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 173110791c984c52bb03c75cffa3bb9d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 20:24:02.633123 25953 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.18.60" port: 32889 } has no permanent_uuid. Determining permanent_uuid...
I20260928 20:24:02.634868 25806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:02.621407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51482 (local address 127.3.18.62:37885)
0928 20:24:02.621448 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:02.621450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:02.623692 (+ 2242us) server_negotiation.cc:407] Connection header received
0928 20:24:02.626482 (+ 2790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:02.626483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:02.626495 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:02.626544 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:02.627215 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:02.627889 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:02.628956 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:02.629268 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:02.629334 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:02.632096 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:02.632097 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:02.632098 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:02.632105 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:02.633709 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:02.634053 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:02.634054 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:02.634054 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:02.634108 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:02.634301 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:02.634302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:02.634302 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:02.634413 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:02.634579 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:02.634626 (+ 47us) server_negotiation.cc:299] Negotiation successful
0928 20:24:02.634699 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
Sep 28 20:24:02 dist-test-slave-b83v krb5kdc[25722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627042, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.60@KRBTEST.COM for kudu/127.3.18.60@KRBTEST.COM
Sep 28 20:24:02 dist-test-slave-b83v krb5kdc[25722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627042, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.18.61@KRBTEST.COM
I20260928 20:24:02.641623 25957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:02.633394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.60:32889 (local address 127.0.0.1:36030)
0928 20:24:02.633460 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:02.633470 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:02.633513 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:02.633655 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:02.633655 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:02.633799 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:02.633838 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:02.633840 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:02.634759 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:02.634760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:02.635550 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:02.635552 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:02.636034 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:02.636035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:02.636097 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:02.636493 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:02.636498 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:02.637913 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:02.640516 (+ 2603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:02.640517 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:02.640518 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:02.640683 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:02.640894 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:02.640894 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:02.640895 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:02.640948 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:02.641270 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:02.641271 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:02.641311 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 20:24:02.641391 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 20:24:02.641494 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 20:24:02.641628 25956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:02.633427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36030 (local address 127.3.18.60:32889)
0928 20:24:02.633467 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:02.633469 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:02.633500 (+ 31us) server_negotiation.cc:407] Connection header received
0928 20:24:02.633572 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:02.633573 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:02.633585 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:02.633643 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:02.633917 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:02.634667 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:02.635639 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:02.635957 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:02.636231 (+ 274us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:02.638007 (+ 1776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:02.638008 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:02.638009 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:02.638075 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:02.640428 (+ 2353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:02.640766 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:02.640767 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:02.640767 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:02.640821 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:02.641013 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:02.641014 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:02.641014 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:02.641203 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:02.641279 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:02.641411 (+ 132us) server_negotiation.cc:299] Negotiation successful
0928 20:24:02.641544 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":18}
I20260928 20:24:02.643603 25882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:02.635405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49760 (local address 127.3.18.61:32869)
0928 20:24:02.635438 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:02.635440 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:02.635449 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:24:02.635508 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:02.635509 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:02.635517 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:02.635561 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:02.636066 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:02.636671 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:02.637342 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:02.637544 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:02.637621 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:02.640135 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:02.640137 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:02.640137 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:02.640145 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:02.642461 (+ 2316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:02.642837 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:02.642838 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:02.642839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:02.642897 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:02.643103 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:02.643104 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:02.643104 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:02.643235 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:02.643402 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:02.643447 (+ 45us) server_negotiation.cc:299] Negotiation successful
0928 20:24:02.643520 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 20:24:02.643608 25953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e120c52c2f5487c8ca16dfce5ce5198: Bootstrap starting.
I20260928 20:24:02.644400 25953 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e120c52c2f5487c8ca16dfce5ce5198: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:02.644711 25953 log.cc:824] T 00000000000000000000000000000000 P 4e120c52c2f5487c8ca16dfce5ce5198: Log is configured to *not* fsync() on all Append() calls
I20260928 20:24:02.644917 3144 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 20:24:02.645495 25953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e120c52c2f5487c8ca16dfce5ce5198: No bootstrap required, opened a new log
I20260928 20:24:02.645831 25777 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "173110791c984c52bb03c75cffa3bb9d"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51482
I20260928 20:24:02.645995 25777 raft_consensus.cc:482] T 00000000000000000000000000000000 P 173110791c984c52bb03c75cffa3bb9d [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 20:24:02.646054 25777 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 173110791c984c52bb03c75cffa3bb9d [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:02.647007 25777 raft_consensus.cc:504] T 00000000000000000000000000000000 P 173110791c984c52bb03c75cffa3bb9d [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "173110791c984c52bb03c75cffa3bb9d" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 37885 } } peers { permanent_uuid: "1b69b1414b084a938036884696a8737c" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 32869 } } peers { permanent_uuid: "4e120c52c2f5487c8ca16dfce5ce5198" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 32889 } }
I20260928 20:24:02.647282 25777 leader_election.cc:288] T 00000000000000000000000000000000 P 173110791c984c52bb03c75cffa3bb9d [CANDIDATE]: Term 1 election: Requested vote from peers 1b69b1414b084a938036884696a8737c (127.3.18.61:32869), 4e120c52c2f5487c8ca16dfce5ce5198 (127.3.18.60:32889)
I20260928 20:24:02.648586 25953 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e120c52c2f5487c8ca16dfce5ce5198 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "173110791c984c52bb03c75cffa3bb9d" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 37885 } } peers { permanent_uuid: "1b69b1414b084a938036884696a8737c" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 32869 } } peers { permanent_uuid: "4e120c52c2f5487c8ca16dfce5ce5198" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 32889 } }
I20260928 20:24:02.648877 25953 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e120c52c2f5487c8ca16dfce5ce5198 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e120c52c2f5487c8ca16dfce5ce5198, State: Initialized, Role: FOLLOWER
I20260928 20:24:02.648993 25953 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e120c52c2f5487c8ca16dfce5ce5198 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "173110791c984c52bb03c75cffa3bb9d" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 37885 } } peers { permanent_uuid: "1b69b1414b084a938036884696a8737c" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 32869 } } peers { permanent_uuid: "4e120c52c2f5487c8ca16dfce5ce5198" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 32889 } }
I20260928 20:24:02.649326 25953 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e120c52c2f5487c8ca16dfce5ce5198 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:24:02.649410 25927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "173110791c984c52bb03c75cffa3bb9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4e120c52c2f5487c8ca16dfce5ce5198"
I20260928 20:24:02.649537 25927 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e120c52c2f5487c8ca16dfce5ce5198 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:02.650913 25927 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4e120c52c2f5487c8ca16dfce5ce5198 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 173110791c984c52bb03c75cffa3bb9d in term 1.
I20260928 20:24:02.651183 25751 leader_election.cc:302] T 00000000000000000000000000000000 P 173110791c984c52bb03c75cffa3bb9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 173110791c984c52bb03c75cffa3bb9d, 4e120c52c2f5487c8ca16dfce5ce5198; no voters:
I20260928 20:24:02.651347 25960 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 173110791c984c52bb03c75cffa3bb9d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:24:02.651522 25960 raft_consensus.cc:686] T 00000000000000000000000000000000 P 173110791c984c52bb03c75cffa3bb9d [term 1 LEADER]: Becoming Leader. State: Replica: 173110791c984c52bb03c75cffa3bb9d, State: Running, Role: LEADER
I20260928 20:24:02.651644 25960 consensus_queue.cc:237] T 00000000000000000000000000000000 P 173110791c984c52bb03c75cffa3bb9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "173110791c984c52bb03c75cffa3bb9d" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 37885 } } peers { permanent_uuid: "1b69b1414b084a938036884696a8737c" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 32869 } } peers { permanent_uuid: "4e120c52c2f5487c8ca16dfce5ce5198" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 32889 } }
I20260928 20:24:02.652208 25960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 173110791c984c52bb03c75cffa3bb9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 173110791c984c52bb03c75cffa3bb9d. Latest consensus state: current_term: 1 leader_uuid: "173110791c984c52bb03c75cffa3bb9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "173110791c984c52bb03c75cffa3bb9d" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 37885 } } peers { permanent_uuid: "1b69b1414b084a938036884696a8737c" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 32869 } } peers { permanent_uuid: "4e120c52c2f5487c8ca16dfce5ce5198" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 32889 } } }
I20260928 20:24:02.652292 25960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 173110791c984c52bb03c75cffa3bb9d [sys.catalog]: This master's current role is: LEADER
W20260928 20:24:02.652685 25991 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4e120c52c2f5487c8ca16dfce5ce5198: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 20:24:02.652742 25991 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 20:24:02.652886 25980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e120c52c2f5487c8ca16dfce5ce5198 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "173110791c984c52bb03c75cffa3bb9d" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 37885 } } peers { permanent_uuid: "1b69b1414b084a938036884696a8737c" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 32869 } } peers { permanent_uuid: "4e120c52c2f5487c8ca16dfce5ce5198" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 32889 } } }
I20260928 20:24:02.652959 25980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e120c52c2f5487c8ca16dfce5ce5198 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 20:24:02.653636 25994 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:24:02.653959 25994 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:24:02.653764 25885 master_runner.cc:186] Error getting master registration for 127.3.18.61:32869: OK, instance_id { permanent_uuid: "1b69b1414b084a938036884696a8737c" instance_seqno: 1790627042432840 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 20:24:02.656203 25885 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 20:24:02.657524 25927 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4e120c52c2f5487c8ca16dfce5ce5198 [term 1 FOLLOWER]: Refusing update from remote peer 173110791c984c52bb03c75cffa3bb9d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:24:02.657866 25960 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 173110791c984c52bb03c75cffa3bb9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e120c52c2f5487c8ca16dfce5ce5198" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 32889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:24:02.659282 25980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e120c52c2f5487c8ca16dfce5ce5198 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 173110791c984c52bb03c75cffa3bb9d. Latest consensus state: current_term: 1 leader_uuid: "173110791c984c52bb03c75cffa3bb9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "173110791c984c52bb03c75cffa3bb9d" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 37885 } } peers { permanent_uuid: "1b69b1414b084a938036884696a8737c" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 32869 } } peers { permanent_uuid: "4e120c52c2f5487c8ca16dfce5ce5198" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 32889 } } }
I20260928 20:24:02.659401 25980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e120c52c2f5487c8ca16dfce5ce5198 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 20:24:02.663720 25882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:02.648491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49768 (local address 127.3.18.61:32869)
0928 20:24:02.648531 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:02.648533 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:02.648830 (+ 297us) server_negotiation.cc:407] Connection header received
0928 20:24:02.648988 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:02.648989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:02.649000 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:02.649030 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:02.650063 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:02.650803 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:02.652853 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:02.653185 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:02.653229 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:02.658109 (+ 4880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:02.658110 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:02.658111 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:02.658119 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:02.660415 (+ 2296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:02.660999 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:02.661000 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:02.661001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:02.661073 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:02.661571 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:02.661572 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:02.661573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:02.661716 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:02.661813 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:02.663528 (+ 1715us) server_negotiation.cc:299] Negotiation successful
0928 20:24:02.663617 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 20:24:02.667248 25956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:02.648686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36040 (local address 127.3.18.60:32889)
0928 20:24:02.648742 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:02.648745 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:02.652403 (+ 3658us) server_negotiation.cc:407] Connection header received
0928 20:24:02.652426 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:02.652427 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:02.652438 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:02.652486 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:02.654400 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:02.655089 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:02.657288 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:02.657648 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:02.657710 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:02.662584 (+ 4874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:02.662586 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:02.662586 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:02.662595 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:02.664612 (+ 2017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:02.665380 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:02.665382 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:02.665382 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:02.665472 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:02.665859 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:02.665860 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:02.665860 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:02.666086 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:02.666840 (+ 754us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:02.666911 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 20:24:02.667053 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 20:24:02.669250 25992 sys_catalog.cc:455] T 00000000000000000000000000000000 P 173110791c984c52bb03c75cffa3bb9d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "173110791c984c52bb03c75cffa3bb9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "173110791c984c52bb03c75cffa3bb9d" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 37885 } } peers { permanent_uuid: "1b69b1414b084a938036884696a8737c" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 32869 } } peers { permanent_uuid: "4e120c52c2f5487c8ca16dfce5ce5198" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 32889 } } }
I20260928 20:24:02.669358 25992 sys_catalog.cc:458] T 00000000000000000000000000000000 P 173110791c984c52bb03c75cffa3bb9d [sys.catalog]: This master's current role is: LEADER
I20260928 20:24:02.669822 25878 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.18.60" port: 32889 } attempt: 3
I20260928 20:24:02.669889 25980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e120c52c2f5487c8ca16dfce5ce5198 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "173110791c984c52bb03c75cffa3bb9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "173110791c984c52bb03c75cffa3bb9d" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 37885 } } peers { permanent_uuid: "1b69b1414b084a938036884696a8737c" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 32869 } } peers { permanent_uuid: "4e120c52c2f5487c8ca16dfce5ce5198" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 32889 } } }
I20260928 20:24:02.669965 25980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e120c52c2f5487c8ca16dfce5ce5198 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 20:24:02.670285 25997 mvcc.cc:205] Tried to move back new op lower bound from 7334408366722322432 to 7334408366702297088. Current Snapshot: MvccSnapshot[applied={T|T < 7334408366722322432}]
I20260928 20:24:02.674242 25806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:02.653234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51498 (local address 127.3.18.62:37885)
0928 20:24:02.653270 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:02.653272 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:02.665257 (+ 11985us) server_negotiation.cc:407] Connection header received
0928 20:24:02.665353 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:02.665354 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:02.665364 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:02.665394 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:02.665951 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:02.666737 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:02.668079 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:02.668397 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:02.668474 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:02.670614 (+ 2140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:02.670615 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:02.670615 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:02.670623 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:02.672332 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:02.673057 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:02.673058 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:02.673058 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:02.673130 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:02.673466 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:02.673467 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:02.673467 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:02.673600 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:02.673673 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:02.674045 (+ 372us) server_negotiation.cc:299] Negotiation successful
0928 20:24:02.674143 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 20:24:02.676029 25994 catalog_manager.cc:1347] Generated new cluster ID: 383efc10b68a488db511b8b2fcf72688
I20260928 20:24:02.676087 25994 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 20:24:02 dist-test-slave-b83v krb5kdc[25722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627042, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.18.61@KRBTEST.COM for kudu/127.3.18.60@KRBTEST.COM
I20260928 20:24:02.695149 25880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:02.670068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.18.60:32889 (local address 127.0.0.1:36044)
0928 20:24:02.670200 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:02.670211 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:02.670248 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:02.670577 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:02.670579 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:02.670737 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:02.670782 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:02.670784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:02.673068 (+ 2284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:02.673069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:02.673871 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:02.673874 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:02.674609 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:02.674610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:02.674674 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:02.675122 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:02.675128 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:02.679825 (+ 4697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:02.693826 (+ 14001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:02.693828 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:02.693828 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:02.694090 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:02.694416 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:02.694418 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:02.694418 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:02.694498 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:02.694802 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:02.694804 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:02.694859 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 20:24:02.694907 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 20:24:02.695037 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115}
I20260928 20:24:02.695798 25956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:02.670326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36044 (local address 127.3.18.60:32889)
0928 20:24:02.670448 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:02.670450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:02.670457 (+ 7us) server_negotiation.cc:407] Connection header received
0928 20:24:02.670475 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:02.670476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:02.670484 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:02.670533 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:02.671058 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:02.672963 (+ 1905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:02.673971 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:02.674276 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:02.674828 (+ 552us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:02.689599 (+ 14771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:02.689601 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:02.689601 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:02.689611 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:02.693682 (+ 4071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:02.694256 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:02.694257 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:02.694257 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:02.694326 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:02.694578 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:02.694578 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:02.694579 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:02.694717 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:02.695518 (+ 801us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:02.695575 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 20:24:02.695667 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260928 20:24:02.696128 25994 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:24:02.696853 25994 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:24:02.697480 25878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b69b1414b084a938036884696a8737c: Bootstrap starting.
I20260928 20:24:02.698222 25878 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b69b1414b084a938036884696a8737c: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:02.698529 25878 log.cc:824] T 00000000000000000000000000000000 P 1b69b1414b084a938036884696a8737c: Log is configured to *not* fsync() on all Append() calls
I20260928 20:24:02.699608 25878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b69b1414b084a938036884696a8737c: No bootstrap required, opened a new log
I20260928 20:24:02.700501 25994 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 173110791c984c52bb03c75cffa3bb9d: Generated new TSK 0
I20260928 20:24:02.700690 25994 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 20:24:02.703557 25878 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b69b1414b084a938036884696a8737c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "173110791c984c52bb03c75cffa3bb9d" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 37885 } } peers { permanent_uuid: "1b69b1414b084a938036884696a8737c" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 32869 } } peers { permanent_uuid: "4e120c52c2f5487c8ca16dfce5ce5198" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 32889 } }
I20260928 20:24:02.703949 25878 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b69b1414b084a938036884696a8737c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b69b1414b084a938036884696a8737c, State: Initialized, Role: FOLLOWER
I20260928 20:24:02.704180 25878 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b69b1414b084a938036884696a8737c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "173110791c984c52bb03c75cffa3bb9d" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 37885 } } peers { permanent_uuid: "1b69b1414b084a938036884696a8737c" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 32869 } } peers { permanent_uuid: "4e120c52c2f5487c8ca16dfce5ce5198" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 32889 } }
I20260928 20:24:02.704689 26003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b69b1414b084a938036884696a8737c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "173110791c984c52bb03c75cffa3bb9d" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 37885 } } peers { permanent_uuid: "1b69b1414b084a938036884696a8737c" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 32869 } } peers { permanent_uuid: "4e120c52c2f5487c8ca16dfce5ce5198" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 32889 } } }
I20260928 20:24:02.704788 26003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b69b1414b084a938036884696a8737c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 20:24:02.705184 25851 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b69b1414b084a938036884696a8737c [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:02.706223 25851 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1b69b1414b084a938036884696a8737c [term 1 FOLLOWER]: Refusing update from remote peer 173110791c984c52bb03c75cffa3bb9d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:24:02.705209 25852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "173110791c984c52bb03c75cffa3bb9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1b69b1414b084a938036884696a8737c"
I20260928 20:24:02.706596 25960 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 173110791c984c52bb03c75cffa3bb9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b69b1414b084a938036884696a8737c" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 32869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:24:02.705091 25878 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b69b1414b084a938036884696a8737c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:24:02.707315 25852 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1b69b1414b084a938036884696a8737c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 173110791c984c52bb03c75cffa3bb9d in term 1.
I20260928 20:24:02.710546 26003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b69b1414b084a938036884696a8737c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 173110791c984c52bb03c75cffa3bb9d. Latest consensus state: current_term: 1 leader_uuid: "173110791c984c52bb03c75cffa3bb9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "173110791c984c52bb03c75cffa3bb9d" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 37885 } } peers { permanent_uuid: "1b69b1414b084a938036884696a8737c" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 32869 } } peers { permanent_uuid: "4e120c52c2f5487c8ca16dfce5ce5198" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 32889 } } }
I20260928 20:24:02.710721 26003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b69b1414b084a938036884696a8737c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 20:24:02.710799 26003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b69b1414b084a938036884696a8737c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "173110791c984c52bb03c75cffa3bb9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "173110791c984c52bb03c75cffa3bb9d" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 37885 } } peers { permanent_uuid: "1b69b1414b084a938036884696a8737c" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 32869 } } peers { permanent_uuid: "4e120c52c2f5487c8ca16dfce5ce5198" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 32889 } } }
I20260928 20:24:02.710902 26003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b69b1414b084a938036884696a8737c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 20:24:02.716578 26005 mvcc.cc:205] Tried to move back new op lower bound from 7334408366895955968 to 7334408366702297088. Current Snapshot: MvccSnapshot[applied={T|T < 7334408366722322432}]
W20260928 20:24:02.716562 26017 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1b69b1414b084a938036884696a8737c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 20:24:02.716718 26017 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260928 20:24:02.736007 25806 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:24:02.734187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51508 (local address 127.3.18.62:37885)
0928 20:24:02.734240 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:02.734242 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:02.735229 (+ 987us) server_negotiation.cc:407] Connection header received
0928 20:24:02.735341 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:02.735343 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:02.735354 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:02.735386 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:02.735927 (+ 541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51508: BlockingRecv error: recv got EOF from 127.0.0.1:51508 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260928 20:24:02.736835 25882 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:24:02.734600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49784 (local address 127.3.18.61:32869)
0928 20:24:02.734654 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:02.734656 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:02.736272 (+ 1616us) server_negotiation.cc:407] Connection header received
0928 20:24:02.736373 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:02.736375 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:02.736386 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:02.736416 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:02.736757 (+ 341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49784: BlockingRecv error: recv got EOF from 127.0.0.1:49784 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260928 20:24:02.737603 25956 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:24:02.734939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36058 (local address 127.3.18.60:32889)
0928 20:24:02.734985 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:02.734988 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:02.737078 (+ 2090us) server_negotiation.cc:407] Connection header received
0928 20:24:02.737177 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:02.737178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:02.737196 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:02.737226 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:02.737538 (+ 312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36058: BlockingRecv error: recv got EOF from 127.0.0.1:36058 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 20:24:02.740182 25882 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:24:02.739317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49798 (local address 127.3.18.61:32869)
0928 20:24:02.739355 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:02.739357 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:02.739535 (+ 178us) server_negotiation.cc:407] Connection header received
0928 20:24:02.739662 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:02.739664 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:02.739686 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:02.739717 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:02.740117 (+ 400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49798: BlockingRecv error: recv got EOF from 127.0.0.1:49798 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 20:24:02.743482 25882 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:24:02.742652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49812 (local address 127.3.18.61:32869)
0928 20:24:02.742691 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:02.742693 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:02.742925 (+ 232us) server_negotiation.cc:407] Connection header received
0928 20:24:02.743027 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:02.743029 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:02.743040 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:02.743070 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:02.743423 (+ 353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49812: BlockingRecv error: recv got EOF from 127.0.0.1:49812 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 20:24:02.747682 25882 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:24:02.746856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49822 (local address 127.3.18.61:32869)
0928 20:24:02.746896 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:02.746898 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:02.747108 (+ 210us) server_negotiation.cc:407] Connection header received
0928 20:24:02.747217 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:02.747218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:02.747231 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:02.747261 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:02.747622 (+ 361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49822: BlockingRecv error: recv got EOF from 127.0.0.1:49822 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 20:24:02.761118 25882 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:24:02.760119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49838 (local address 127.3.18.61:32869)
0928 20:24:02.760170 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:02.760172 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:02.760467 (+ 295us) server_negotiation.cc:407] Connection header received
0928 20:24:02.760574 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:02.760576 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:02.760590 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:02.760634 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:02.761049 (+ 415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49838: BlockingRecv error: recv got EOF from 127.0.0.1:49838 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260928 20:24:02.771449 25882 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:24:02.770524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49852 (local address 127.3.18.61:32869)
0928 20:24:02.770569 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:02.770572 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:02.770845 (+ 273us) server_negotiation.cc:407] Connection header received
0928 20:24:02.770954 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:02.770955 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:02.770967 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:02.770997 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:02.771378 (+ 381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49852: BlockingRecv error: recv got EOF from 127.0.0.1:49852 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 20:24:02.798895 25882 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:24:02.797920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49862 (local address 127.3.18.61:32869)
0928 20:24:02.797968 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:02.797970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:02.798248 (+ 278us) server_negotiation.cc:407] Connection header received
0928 20:24:02.798361 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:02.798362 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:02.798375 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:02.798408 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:02.798833 (+ 425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49862: BlockingRecv error: recv got EOF from 127.0.0.1:49862 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 20:24:02.833340 25882 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:24:02.832445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49864 (local address 127.3.18.61:32869)
0928 20:24:02.832494 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:02.832497 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:02.832685 (+ 188us) server_negotiation.cc:407] Connection header received
0928 20:24:02.832745 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:02.832746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:02.832760 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:02.832814 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:02.833268 (+ 454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49864: BlockingRecv error: recv got EOF from 127.0.0.1:49864 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 20:24:02.899581 25882 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:24:02.898770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49870 (local address 127.3.18.61:32869)
0928 20:24:02.898870 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:02.898872 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:02.898971 (+ 99us) server_negotiation.cc:407] Connection header received
0928 20:24:02.899050 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:02.899051 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:02.899065 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:02.899138 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:02.899517 (+ 379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49870: BlockingRecv error: recv got EOF from 127.0.0.1:49870 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260928 20:24:03.029680 25882 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:24:03.028852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49884 (local address 127.3.18.61:32869)
0928 20:24:03.028935 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:03.028938 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:03.029040 (+ 102us) server_negotiation.cc:407] Connection header received
0928 20:24:03.029174 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:03.029179 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:03.029191 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:03.029248 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:03.029614 (+ 366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49884: BlockingRecv error: recv got EOF from 127.0.0.1:49884 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 20:24:03.174847 25960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 173110791c984c52bb03c75cffa3bb9d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "173110791c984c52bb03c75cffa3bb9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "173110791c984c52bb03c75cffa3bb9d" member_type: VOTER last_known_addr { host: "127.3.18.62" port: 37885 } } peers { permanent_uuid: "1b69b1414b084a938036884696a8737c" member_type: VOTER last_known_addr { host: "127.3.18.61" port: 32869 } } peers { permanent_uuid: "4e120c52c2f5487c8ca16dfce5ce5198" member_type: VOTER last_known_addr { host: "127.3.18.60" port: 32889 } } }
I20260928 20:24:03.175021 25960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 173110791c984c52bb03c75cffa3bb9d [sys.catalog]: This master's current role is: LEADER
W20260928 20:24:03.287802 25882 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:24:03.287113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49886 (local address 127.3.18.61:32869)
0928 20:24:03.287186 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:03.287188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:03.287195 (+ 7us) server_negotiation.cc:407] Connection header received
0928 20:24:03.287220 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:03.287221 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:03.287230 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:03.287300 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:03.287714 (+ 414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49886: BlockingRecv error: recv got EOF from 127.0.0.1:49886 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 20:24:03.653208 25991 catalog_manager.cc:1259] Loaded cluster ID: 383efc10b68a488db511b8b2fcf72688
I20260928 20:24:03.653334 25991 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4e120c52c2f5487c8ca16dfce5ce5198: loading cluster ID for follower catalog manager: success
I20260928 20:24:03.654206 25991 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4e120c52c2f5487c8ca16dfce5ce5198: acquiring CA information for follower catalog manager: success
I20260928 20:24:03.654556 25991 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4e120c52c2f5487c8ca16dfce5ce5198: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 20:24:03.717152 26017 catalog_manager.cc:1259] Loaded cluster ID: 383efc10b68a488db511b8b2fcf72688
I20260928 20:24:03.717253 26017 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1b69b1414b084a938036884696a8737c: loading cluster ID for follower catalog manager: success
I20260928 20:24:03.718132 26017 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1b69b1414b084a938036884696a8737c: acquiring CA information for follower catalog manager: success
I20260928 20:24:03.718449 26017 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1b69b1414b084a938036884696a8737c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 20:24:03.804648 26087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:03.801004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49890 (local address 127.3.18.61:32869)
0928 20:24:03.801155 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:03.801157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:03.801254 (+ 97us) server_negotiation.cc:407] Connection header received
0928 20:24:03.801354 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:03.801355 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:03.801415 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:03.801498 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:24:03.801668 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:03.802340 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:03.803467 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:03.803847 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:03.803932 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:03.804193 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:24:03.804347 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:24:03.804445 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:03.804520 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 20:24:03.804549 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":66,"threads_started":1}
I20260928 20:24:03.809345 26093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:03.806598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36074 (local address 127.3.18.60:32889)
0928 20:24:03.806708 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:03.806710 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:03.806743 (+ 33us) server_negotiation.cc:407] Connection header received
0928 20:24:03.806812 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:03.806812 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:03.806866 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:03.806934 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:24:03.807013 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:03.807655 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:03.808518 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:03.808771 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:03.808830 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:03.808997 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:24:03.809105 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:24:03.809166 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:03.809202 (+ 36us) server_negotiation.cc:299] Negotiation successful
0928 20:24:03.809219 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":50,"threads_started":1}
I20260928 20:24:03.814393 26087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:03.811252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49894 (local address 127.3.18.61:32869)
0928 20:24:03.811412 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:03.811414 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:03.811423 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:24:03.811453 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:03.811454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:03.811465 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:03.811525 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:24:03.811642 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:03.812441 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:03.813575 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:03.813891 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:03.813928 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:03.814119 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:24:03.814152 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:24:03.814211 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:03.814270 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 20:24:03.814294 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260928 20:24:03.814429 26100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:03.811310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51518 (local address 127.3.18.62:37885)
0928 20:24:03.811606 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:03.811608 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:03.811710 (+ 102us) server_negotiation.cc:407] Connection header received
0928 20:24:03.811800 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:03.811801 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:03.811851 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:03.811927 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:24:03.812103 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:03.812723 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:03.813557 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:03.813803 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:03.813872 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:03.814038 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:24:03.814155 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:24:03.814214 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:03.814310 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 20:24:03.814340 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":57,"threads_started":1}
I20260928 20:24:03.814652 26093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:03.811464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36078 (local address 127.3.18.60:32889)
0928 20:24:03.811568 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:03.811570 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:03.811852 (+ 282us) server_negotiation.cc:407] Connection header received
0928 20:24:03.811909 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:03.811910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:03.811923 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:03.811989 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:24:03.812158 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:03.812781 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:03.813806 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:03.814084 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:03.814154 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:03.814389 (+ 235us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:24:03.814419 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:24:03.814446 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:03.814554 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 20:24:03.814575 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 20:24:03.816919 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 25735
I20260928 20:24:03.822527 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 25810
I20260928 20:24:03.827687 3144 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2yfqO7/build/release/bin/kudu with pid 25885
2026-09-28T20:24:03Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2892 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5732 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (279584 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST